I am looking to make a combination JMeter extensions that satisfy a few criteria:
Open files at the beginning of the test plan/thread group. One of the files in an excel file used for input. I have the code to read it using apache poi via including the apache tika jar in the jmeter lib folder. The input should then be used in the threads as variables just like it's done with the CSV Data Set Config test element.
Aggregate all results at the end of the test plan/thread group to do calculations on the set of all results.
For #1, Maybe it is possible to do this by extending a config element but I haven't seen how to do this yet. I am also unsure of how to mimic the behaviour of CSV Data Set Config.
For #2 the purpose is to send the final information extracted from the results to a server so saving results to a file is not optimal. The View Results Tree or the View Results in Table elements both create a report of all results so it seems it should be possible to do this.
Edit:
How to achieve the above?
Assuming your question as 'How to achieve the above?'
For #1:
First of, I believe it is much easier/simpler (because Simple is better than complex) to get the excel/app to provide csv file for jmeter to consume it using CSV Data Set Config. I mean, write the reading logic somewhere else which will feed into jmeter's test data file. Another option would be to write a JSR223 sampler in a set up threadgroup to read the excel and produce the CSV.
But if you need it anyhow, you will need to write a custom plugin which will inherit from ConfigTestElement and will need to implement TestBean and LoopIterationListener interfaces. Good place to start is here. And the code for CSV Data Set Config is here.
For #2:
If you need POST the result file to server then you can use a tearDown thread group in jmeter which will pick up the file at the end of the test and do a HTTP post request using HTTP Request sampler.
Hope I gave you some direction.
Related
I have a scenario where I need to collect the data from different testng emailable-html report and make it as single one as look like same as testng-emailable report.
I didn't get any idea about this like how can I achieve it. I just want to create a seperate java project which will read the different html report and will make it final consolidate report as look like testng-emailable report.
Please give me some input on this, like Is there any listener can create the html template and accept the input data from html file.
I didn't look working code. Just looking some input to kick start this requirement.
TestNG is open source software, so you could build your solution from the original code. You start with the EmailableReporter class. You collect the different results as XML files (testng-results.xml) and merge them in your code to get proper input for EmailableReporter. This will generate your desired output.
I have 50k machines and each machine is having a unique id.
every 10 seconds machine will send a file in machine_feed directory located in ftp server.Not all files are received at same time.
Machine will create file with it's id name.
I need to process all received files. If file is not processed in short time then machine will send new file that will override existing file and i will loose existing data.
My Solution is
I have created spring boot application contains one scheduler that execute every 1 millisecond, that will rename received file and will copy it to processing dir. current date time will be appended to each file.
I have one more job written in apache camel that will poll received file from processnig location for every 500 milisecond and will process it and insert data in DB.if error is received then it will move file in error dir.
File is not big. It contains only one line of information.
Issue is if files are less then it is doing great job. If files are increasing then though file is valid it is moving in error folder.
when camel is polling file then found zero length file and after that file is copied to error directory then it contains valid data. Some how camel is polling file that is not copied completely.
Anyone know good solution for this problem?.
Thanks in advance.
I've faced a similar problem before but I used a slightly different set of tools...
I would recommend taking a look at Apache Flume - it is a lightweight java process. This is what I used in my situation. The documentation is pretty decent so you should be able to find your way but I just thought of giving a brief introduction anyway just to get you started.
Flume has 3 main components and each of these can be configured in various ways:
Source - The component responsible for sourcing the data
Channel - Buffer component
Sink - This would represent the destination where the data needs to land
There are other optional components as well such as Interceptor - which is primarily useful for intercepting the flow and carrying out basic filtering, transformations etc.
There is wide variety of options to choose from for each of these but if none of the ones available suit your use case - you could write your own component.
Now, for your situation - following are a couple of options I could think of:
Since your file location needs almost continuous monitoring, you might want to use Flume's Spooling Directory Source that would continuously watch your machine_feed directory and pick it up as soon as the file arrives (You could choose to alter the name yourself before the file gets overwritten).
So, the idea is to pick up the file and hand it over to the processing directory and then carry on with the processing with Apache Camel as you are already doing it.
The other option would be (and this is the one I would recommend considering) - Do everything in one Flume agent.
Your flume set-up could look like this:
Spooling Directory Source
One of the interceptors (Optional: for your processing before inserting the data into the DB. If none of the available options are suitable - you could even write your own custom interceptor)
One of the channels (Memory channel - May be...)
Lastly, one of the sinks (This might just need to be a custom sink in your case for landing the data in a DB)
If you do need to write up a custom component (an interceptor or a sink), you could just look at the source code of one the default components for reference. Here's the link to the source code repository.
I understand that I've gone in a slightly different tangent by suggesting a new tool altogether but this worked magically for me as the tool is a very light weight tool with a fairly straightforward set up and configuration.
I hope this helps.
I'm working on a scenario with two pipelines using DataFlow in Google Cloud:
Pipeline A runs in streaming mode continuously creating files in Google-Storage based on hourly-windows and some sharding like this:
data.apply(TextIO.write().to(resource.getCurrentDirectory())
.withFilenamePolicy(new PerWindowFiles(prefix))
.withWindowedWrites()
.withNumShards(42));
Pipeline B works in batching mode loading those files regularily for further processing, e.g. every hour.
Here's the Problem: which files can the pipeline B savely load from GS?
All of them -> probably not a good idea in case A is not done writing some of them and we'll get corrupted files.
based on time (like load only the files that are at least 2h old) -> will also cause issues in case A is late
some way of creating "done"-flags in A which tell B which files are done.
somehow get notified when a window's final pane is done processing -> haven't found a way to do that.
I would like the third approach, but couldn't find a way of determining when TextIO is actually done writing a file without waiting for the pipeline to finish.
The Writer of TextIO does not return another PCollection. One way would be to override the finalize method of the FileBasedSink.WriteOperation which is created somewhere inside TextIO and requires copying the whole class and eventually building a custom Sink. This is overkill in my opinion.
Anyone has ideas for an easier solution or experience how to achieve this?
TextIO.write() will write data to temporary files, and then atomically rename each successfully written temporary file to its final location. You can safely use the files matching your "prefix" in pipeline B, because temporary files will be named in a way that does not match the prefix (we explicitly accounted for your use case when deciding how to name temporary files), so all files seen by pipeline B will be complete.
Alternatively, we're about to add (link to pull request) a version of TextIO.read() that continuously ingests new files in streaming mode; when that's ready, you can use it in your pipeline B. See also http://s.apache.org/textio-sdf and the linked JIRAs.
I have a directory with files that cannot be removed because they are used by other applications or have read only properties. This means that I can't move or delete the files like Mule does as a natural file tracking system. In order to process these files through Mule once they arrive or when they get updated without deleting/moving them from the original directory I need some sort of custom tracking. To do this I think I need to add some rules and be able to track files that are:
New files
Processed files
Updated files
For this, I thought of having a log file in the same directory that would track each file by name and date modified, but I'm not sure if this is the correct way of doing this. I would need to be able to write and read this log file and compare its content with current files in the directory in order to determine which files are new or updated. This seems to be a bit too complicated and requires me to add quite a bit of programming (maybe as groovy scripts or overriding some methods).
Is there any other simpler way to do this on Mule? If not, how should I start tackling this problem? I'm guessing I can write some java to talk to File EndPoint.
As Victor Romero pointed out, Idempotent Filter does the trick. I tried two types of Idempotent Filter to see which one works best: Idempotent Message Filter and Idempotent Secure Hash Message Filter. Both of them did the job, however I ended up using Idempotent Message Filter (no Hash) to log timestamp and filename in the simple-text-file-store.
Just after the File inbound-endpoint:
<idempotent-message-filter idExpression="#[message.inboundProperties.originalFilename+'-'+message.inboundProperties.timestamp]" storePrefix="prefix" doc:name="Idempotent Message">
<simple-text-file-store name="uniqueProcessedMessages" directory="C:\yourDirectory"/>
</idempotent-message-filter>
Only new or modified files for the purposes of my process would pass through. However Idempotent Secure Hash Message Filter should do a better job at identifying different files.
I have an excel file that pulls in data via data connection from bunch of CSV files. The CSV files are generated every now so often by a JAVA program.
Is it possible to refresh the data too via. JAVA program? I see JXCELAPI and JOI are there, but briefly looking at their documentation doesn't indicate my use case is even possible.
In short, I need API that could achieve this effect: clicking on menu Data->Refresh All.
Thanks,
_Madhu
You could try XLLoop. This lets your spreadsheet talk directly to a java server (or a number of other languages) via function calls.
So you can have your spreadsheet call eg GetMyData("somedata") and it will load the data directly from your server whenever you re-calc (ie. Shift-F9).
BTW, I work on the project so let me know if you have any questions.
This tutorial sounds like it might help: Accessing Excel from Java
You could also try Obba which is another solution to access a Java library via Excel cell functions (UDFs)...
However, what you describe could also be done by a very small vb/vba macro which checks/polls for modification of these CVS files. I don't know if this is suitable in your situation, but there is an event listener for that: http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.changed.aspx#Y0