accumulo - batchscanner: one result per range - java

So my general question is "Is it possible to have an Accumulo BatchScanner only pull back the first result per Range I give it?"
Now some details about my use case as there may be a better way to approach this anyway. I have data that represent messages from different systems. There can be different types of messages. My users want to be able to ask the system questions, such as "give me the most recent message of a certain type as of a certain time for all these systems".
My table layout looks like this
rowid: system_name, family: message_type, qualifier: masked_timestamp, value: message_text
The idea is that the user gives me a list of systems they care about, the type of message, and a certain timestamp. I used masked timestamp so that the table sorts most recent first. That way when I scan for a timestamp, the first result is the most recent prior to that time. I am using a BatchScanner because I have multiple systems I am searching for per query. Can I make the BatchScanner only fetch the first result for each Range? I can't specify a specific key because the most recent may not match the datetime given by the user.
Currently, I am using the BatchScanner and ignoring all but the first result per Key. It works right now, but it seems like a waste to pull back all the data for a specific system/type over the network when I only care about the first result per system/type.
EDIT
My attempt using the FirstEntryInRowIterator
#Test
public void testFirstEntryIterator() throws Exception
{
Connector connector = new MockInstance("inst").getConnector("user", new PasswordToken("password"));
connector.tableOperations().create("testing");
BatchWriter writer = writer(connector, "testing");
writer.addMutation(mutation("row", "fam", "qual1", "val1"));
writer.addMutation(mutation("row", "fam", "qual2", "val2"));
writer.addMutation(mutation("row", "fam", "qual3", "val3"));
writer.close();
Scanner scanner = connector.createScanner("testing", new Authorizations());
scanner.addScanIterator(new IteratorSetting(50, FirstEntryInRowIterator.class));
Key begin = new Key("row", "fam", "qual2");
scanner.setRange(new Range(begin, begin.followingKey(PartialKey.ROW_COLFAM_COLQUAL)));
int numResults = 0;
for (Map.Entry<Key, Value> entry : scanner)
{
Assert.assertEquals("qual2", entry.getKey().getColumnQualifier().toString());
numResults++;
}
Assert.assertEquals(1, numResults);
}
My goal is that the returned entry will be the ("row", "fam", "qual2", "val2") but I get 0 results. It almost seems like the Iterator is being applied before the Range maybe? I haven't dug into this yet.

This sounds like a good use case for using one of Accumulo's SortedKeyValueIterators, specifically the FirstEntryInRowIterator (contained in the accumulo-core artifact).
Create an IteratorSetting with the FirstEntryInRowIterator and add it to your BatchScanner. This will return the first Key/Value in that system_name, and then stop avoiding the overhead of your client ignoring all other results.
A quick modification of the FirstEntryInRowIterator might get you what you want:
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.accumulo.core.iterators;
import java.io.IOException;
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.data.ByteSequence;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.PartialKey;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.hadoop.io.Text;
public class FirstEntryInRangeIterator extends SkippingIterator implements OptionDescriber {
// options
static final String NUM_SCANS_STRING_NAME = "scansBeforeSeek";
// iterator predecessor seek options to pass through
private Range latestRange;
private Collection<ByteSequence> latestColumnFamilies;
private boolean latestInclusive;
// private fields
private Text lastRowFound;
private int numscans;
/**
* convenience method to set the option to optimize the frequency of scans vs. seeks
*/
public static void setNumScansBeforeSeek(IteratorSetting cfg, int num) {
cfg.addOption(NUM_SCANS_STRING_NAME, Integer.toString(num));
}
// this must be public for OptionsDescriber
public FirstEntryInRangeIterator() {
super();
}
public FirstEntryInRangeIterator(FirstEntryInRangeIterator other, IteratorEnvironment env) {
super();
setSource(other.getSource().deepCopy(env));
}
#Override
public SortedKeyValueIterator<Key,Value> deepCopy(IteratorEnvironment env) {
return new FirstEntryInRangeIterator(this, env);
}
#Override
public void init(SortedKeyValueIterator<Key,Value> source, Map<String,String> options, IteratorEnvironment env) throws IOException {
super.init(source, options, env);
String o = options.get(NUM_SCANS_STRING_NAME);
numscans = o == null ? 10 : Integer.parseInt(o);
}
// this is only ever called immediately after getting "next" entry
#Override
protected void consume() throws IOException {
if (finished == true || lastRowFound == null)
return;
int count = 0;
while (getSource().hasTop() && lastRowFound.equals(getSource().getTopKey().getRow())) {
// try to efficiently jump to the next matching key
if (count < numscans) {
++count;
getSource().next(); // scan
} else {
// too many scans, just seek
count = 0;
// determine where to seek to, but don't go beyond the user-specified range
Key nextKey = getSource().getTopKey().followingKey(PartialKey.ROW);
if (!latestRange.afterEndKey(nextKey))
getSource().seek(new Range(nextKey, true, latestRange.getEndKey(), latestRange.isEndKeyInclusive()), latestColumnFamilies, latestInclusive);
else {
finished = true;
break;
}
}
}
lastRowFound = getSource().hasTop() ? getSource().getTopKey().getRow(lastRowFound) : null;
}
private boolean finished = true;
#Override
public boolean hasTop() {
return !finished && getSource().hasTop();
}
#Override
public void seek(Range range, Collection<ByteSequence> columnFamilies, boolean inclusive) throws IOException {
// save parameters for future internal seeks
latestRange = range;
latestColumnFamilies = columnFamilies;
latestInclusive = inclusive;
lastRowFound = null;
super.seek(range, columnFamilies, inclusive);
finished = false;
if (getSource().hasTop()) {
lastRowFound = getSource().getTopKey().getRow();
if (range.beforeStartKey(getSource().getTopKey()))
consume();
}
}
#Override
public IteratorOptions describeOptions() {
String name = "firstEntry";
String desc = "Only allows iteration over the first entry per range";
HashMap<String,String> namedOptions = new HashMap<String,String>();
namedOptions.put(NUM_SCANS_STRING_NAME, "Number of scans to try before seeking [10]");
return new IteratorOptions(name, desc, namedOptions, null);
}
#Override
public boolean validateOptions(Map<String,String> options) {
try {
String o = options.get(NUM_SCANS_STRING_NAME);
if (o != null)
Integer.parseInt(o);
} catch (Exception e) {
throw new IllegalArgumentException("bad integer " + NUM_SCANS_STRING_NAME + ":" + options.get(NUM_SCANS_STRING_NAME), e);
}
return true;
}
}

Related

Not able to update page number in TOC of document using docx4j

While generating Table of content , i am not able to update page number.
It shows the exception as "Encountered broken bookmarks; not configured to remediate."
I used the below code to update TOC ..
TocGenerator tocGenerator = new TocGenerator(wordMLPackage);
tocGenerator.generateToc( 0, " TOC \o \"1-3\" \h \z \u ", false);
tocGenerator.updateToc( false);
The message is coming from here:
/**
* Calculate page numbers
*
* #return
* #throws TocException
*/
private Map<String, Integer> getPageNumbersMap() throws TocException {
// #since 6.1, check bookmarks are ok first
// what to do if not ok?
// - default behaviour is to fail
// - but can be configured to remediate:
boolean remediate = Docx4jProperties.getProperty("docx4j.toc.BookmarksIntegrity.remediate", false);
//
BookmarksIntegrity bm = new BookmarksIntegrity();
StringWriter sw = new StringWriter();
bm.setWriter(sw);
BookmarksStatus result = null;
try {
// Checks are performed on all bookmarks, not just those with
// a name of the form "_Toc*". We don't check for missing _Toc bookmarks.
result = bm.check(wordMLPackage.getMainDocumentPart(), remediate);
} catch (Exception e) { /* won't happen */}
if (result==BookmarksStatus.BROKEN) {
throw new TocException("Encountered broken bookmarks; not configured to remediate. \n" + sw.toString());
}
if (Docx4J.pdfViaFO()) {
return getPageNumbersMapViaFOP();
} else {
// recommended
return getPageNumbersMapViaService();
}
}
Please note that unless you are an existing licensee of Plutext's PDF Converter service, you won't be able to use getPageNumbersMapViaService().
For a possible alternative approach, please see https://www.docx4java.org/blog/2020/03/documents4j-for-toc-update/ which builds upon https://www.docx4java.org/blog/2020/03/documents4j-for-pdf-output/

Mapping characters to keycodes for international keysets

so I built a pi zero keyboard emulator as mentioned here:
https://www.rmedgar.com/blog/using-rpi-zero-as-keyboard-setup-and-device-definition
I make it type text that it reads from a local text-file (everything developed in java - for reasons :) ).
My problem now is that the configured keysets on the various computers my pi zero is attached to differ very much (german, english, french, ...). Depending on the computer this leads to several typing mistakes (e.g., z instead of y).
So I now built some "translation tables" that map characters to the keycodes fitting to the computer. Such a table looks like this:
public scancodes_en_us() {
//We have (Character, (scancode, modifier))
table.put("a",Pair.create("4","0"));
table.put("b",Pair.create("5","0"));
table.put("c",Pair.create("6","0"));
table.put("d",Pair.create("7","0"));
table.put("e",Pair.create("8","0"));
table.put("f",Pair.create("9","0"));
table.put("g",Pair.create("10","0"));
table.put("h",Pair.create("11","0"));
table.put("i",Pair.create("12","0"));
table.put("j",Pair.create("13","0"));
table.put("k",Pair.create("14","0"));
table.put("l",Pair.create("15","0"));
table.put("m",Pair.create("16","0"));
table.put("n",Pair.create("17","0"));
table.put("o",Pair.create("18","0"));
table.put("p",Pair.create("19","0"));
table.put("q",Pair.create("20","0"));
table.put("r",Pair.create("21","0"));
table.put("s",Pair.create("22","0"));
table.put("t",Pair.create("23","0"));
table.put("u",Pair.create("24","0"));
table.put("v",Pair.create("25","0"));
table.put("w",Pair.create("26","0"));
table.put("x",Pair.create("27","0"));
table.put("y",Pair.create("28","0"));
table.put("z",Pair.create("29","0"));
table.put("A",Pair.create("4","2"));
table.put("B",Pair.create("5","2"));
table.put("C",Pair.create("6","2"));
table.put("D",Pair.create("7","2"));
table.put("E",Pair.create("8","2"));
table.put("F",Pair.create("9","2"));
table.put("G",Pair.create("10","2"));
table.put("H",Pair.create("11","2"));
table.put("I",Pair.create("12","2"));
table.put("J",Pair.create("13","2"));
table.put("K",Pair.create("14","2"));
table.put("L",Pair.create("15","2"));
table.put("M",Pair.create("16","2"));
table.put("N",Pair.create("17","2"));
table.put("O",Pair.create("18","2"));
table.put("P",Pair.create("19","2"));
table.put("Q",Pair.create("20","2"));
table.put("R",Pair.create("21","2"));
table.put("S",Pair.create("22","2"));
table.put("V",Pair.create("25","2"));
table.put("W",Pair.create("26","2"));
table.put("X",Pair.create("27","2"));
table.put("Y",Pair.create("28","2"));
table.put("Z",Pair.create("29","2"));
table.put("1",Pair.create("30","0"));
table.put("2",Pair.create("31","0"));
table.put("5",Pair.create("34","0"));
table.put("6",Pair.create("35","0"));
table.put("7",Pair.create("36","0"));
table.put("8",Pair.create("37","0"));
table.put("9",Pair.create("38","0"));
table.put("0",Pair.create("39","0"));
table.put("!",Pair.create("30","2"));
table.put("#",Pair.create("31","2"));
table.put("#",Pair.create("32","2"));
table.put("$",Pair.create("33","2"));
table.put("%",Pair.create("34","2"));
table.put("^",Pair.create("35","2"));
table.put("&",Pair.create("36","2"));
table.put("*",Pair.create("37","2"));
table.put("(",Pair.create("38","2"));
table.put(")",Pair.create("39","2"));
table.put(" ",Pair.create("44","0"));
table.put("-",Pair.create("45","0"));
table.put("=",Pair.create("46","0"));
table.put("[",Pair.create("47","0"));
table.put("]",Pair.create("48","0"));
table.put("\\",Pair.create("49","0"));
table.put(";",Pair.create("51","0"));
table.put("'",Pair.create("52","0"));
table.put("`",Pair.create("53","0"));
table.put(",",Pair.create("54","0"));
table.put(".",Pair.create("55","0"));
table.put("/",Pair.create("56","0"));
table.put("_",Pair.create("45","2"));
table.put("+",Pair.create("46","2"));
table.put("{",Pair.create("47","2"));
table.put("}",Pair.create("48","2"));
table.put("|",Pair.create("49","2"));
table.put(":",Pair.create("51","2"));
table.put("\"",Pair.create("52","2"));
table.put("~",Pair.create("53","2"));
table.put("<",Pair.create("54","2"));
table.put(">",Pair.create("55","2"));
table.put("?",Pair.create("56","2"));
Having such a table for many different keyboard layouts is a pain. Is there some more clever version to map a character to the scancode for a specific keyboard layout?
If not - is there some kind of archive where I can find such a character to scancode mapping for many different keyboard layouts?
Thank you very much
Look at how localization works, they all share the same approach: Create a special version for each localization as a property file, then have an abstract class to load the property based on locale.
You will develop a loader class like this:
public scancodes(Locale locale) {
// load locale property file or download if missing
// read the property and store to the table
ResourceBundle scanCodes = ResourceBundle.getBundle("codes",locale);
}
And your codes_locale looks like:
codes_de.properties
a=4,0
b=5,0
By doing this, you separate the locale specific character with your logic code, and you don't need to bundle all keyboards in side your app. You can download them as needed.
You can access a tutorial here
If I understood what you are trying to do correctly then you don't have to map anything at all, just use a pre-made format (like unicode which works for all languages I know of), just send a char code and translate it to it's matching char.
Example file reader - char interpreter:
JFileChooser fc = new JFileChooser();
fc.setFileSelectionMode(JFileChooser.FILES_ONLY);
fc.showOpenDialog(null);
File textFile = fc.getSelectedFile();
if(textFile.getName().endsWith(".txt")) {
System.out.println(textFile.getAbsolutePath());
FileInputStream input = new FileInputStream(textFile);
BufferedReader reader = new BufferedReader(new InputStreamReader(input, "UNICODE"));
char[] buffer = new char[input.available() / 2 - 1];
System.out.println("Bytes left: " + input.available());
int read = reader.read(buffer);
System.out.println("Read " + read + " characters");
for(int i = 0; i < read; i++) {
System.out.print("The letter is: " + buffer[i]);
System.out.println(", The key code is: " + (int) buffer[i]);
}
}
you can later use the key code to emulate a key press on your computer
For scan-code mappings you can visit following sites:
Keyboard scancodes
Scan Codes Demystified
My solution is to determine the list of keycode on runtime, it'll save you a lot of caffeine and headache
package test;
import java.util.HashMap;
import java.util.Map;
import javax.swing.KeyStroke;
public class Keycode {
/**
* List of chars, can be stored in file
* #return
*/
public String getCharsets() {
return "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSVWXYZ12567890!##$%^&*() -=[]\\;'`,./_+{}|:\\~<>?";
}
/**
* Determines the keycode on runtime
* #return
*/
public Map<Character, Integer> getScancode() {
Map<Character, Integer> table = new HashMap<>();
String charsets = this.getCharsets();
for( int index = 0 ; index < charsets.length() ; index++ ) {
Character currentChar = charsets.charAt(index);
KeyStroke keyStroke = KeyStroke.getKeyStroke(currentChar.charValue(), 0);
// only for example i've used Map, but you should populate it by your table
// table.put("a",Pair.create("4","0"));
table.put(currentChar, keyStroke.getKeyCode());
}
return table;
}
public static void main(String[] args) {
System.out.println(new Keycode().getScancode());
}
}

Migration from dcm4che2 to dcm4che3

I have used below mentioned API of dcm4che2 from this repository http://www.dcm4che.org/maven2/dcm4che/ in my java project.
dcm4che-core-2.0.29.jar
org.dcm4che2.data.DicomObject
org.dcm4che2.io.StopTagInputHandler
org.dcm4che2.data.BasicDicomObject
org.dcm4che2.data.UIDDictionary
org.dcm4che2.data.DicomElement
org.dcm4che2.data.SimpleDcmElement
org.dcm4che2.net.service.StorageCommitmentService
org.dcm4che2.util.CloseUtils
dcm4che-net-2.0.29.jar
org.dcm4che2.net.CommandUtils
org.dcm4che2.net.ConfigurationException
org.dcm4che2.net.NetworkApplicationEntity
org.dcm4che2.net.NetworkConnection
org.dcm4che2.net.NewThreadExecutor
org.dcm4che3.net.service.StorageService
org.dcm4che3.net.service.VerificationService
Currently i want to migrate to dcm4che3 but, above listed API is not found in dcm4che3 which i have downloaded from this repository http://sourceforge.net/projects/dcm4che/files/dcm4che3/
Could you please guide me for alternate approach?
As you have already observed, the BasicDicomObject is history -- alongside quite a few others.
The new "Dicom object" is Attributes -- an object is a collection of attributes.
Therefore, you create Attributes, populate them with the tags you need for RQ-behaviour (C-FIND, etc) and what you get in return is another Attributes object from which you pull the tags you want.
In my opinion, dcm4che 2.x was vague on the subject of dealing with individual value representations. dcm4che 3.x is quite a bit clearer.
The migration demands a rewrite of your code regarding how you query and how you treat individual tags. On the other hand, dcm4che 3.x makes the new code less convoluted.
On request, I have added the initial setup of a connection to some service class provider (SCP):
// Based on org.dcm4che:dcm4che-core:5.25.0 and org.dcm4che:dcm4che-net:5.25.0
import org.dcm4che3.data.*;
import org.dcm4che3.net.*;
import org.dcm4che3.net.pdu.AAssociateRQ;
import org.dcm4che3.net.pdu.PresentationContext;
import org.dcm4che3.net.pdu.RoleSelection;
import org.dcm4che3.net.pdu.UserIdentityRQ;
// Client side representation of the connection. As a client, I will
// not be listening for incoming traffic (but I could choose to do so
// if I need to transfer data via MOVE)
Connection local = new Connection();
local.setHostname("client.on.network.com");
local.setPort(Connection.NOT_LISTENING);
// Remote side representation of the connection
Connection remote = new Connection();
remote.setHostname("pacs.on.network.com");
remote.setPort(4100);
remote.setTlsProtocols(local.getTlsProtocols());
remote.setTlsCipherSuites(local.getTlsCipherSuites());
// Calling application entity
ApplicationEntity ae = new ApplicationEntity("MeAsAServiceClassUser".toUpperCase());
ae.setAETitle("MeAsAServiceClassUser");
ae.addConnection(local); // on which we may not be listening
ae.setAssociationInitiator(true);
ae.setAssociationAcceptor(false);
// Device
Device device = new Device("MeAsAServiceClassUser".toLowerCase());
device.addConnection(local);
device.addApplicationEntity(ae);
// Configure association
AAssociateRQ rq = new AAssociateRQ();
rq.setCallingAET("MeAsAServiceClassUser");
rq.setCalledAET("NameThatIdentifiesTheProvider"); // e.g. "GEPACS"
rq.setImplVersionName("MY-SCU-1.0"); // Max 16 chars
// Credentials (if appropriate)
String username = "username";
String passcode = "so secret";
if (null != username && username.length() > 0 && null != passcode && passcode.length() > 0) {
rq.setUserIdentityRQ(UserIdentityRQ.usernamePasscode(username, passcode.toCharArray(), true));
}
Example, pinging the PACS (using the setup above):
String[] TRANSFER_SYNTAX_CHAIN = {
UID.ExplicitVRLittleEndian,
UID.ImplicitVRLittleEndian
};
// Define transfer capabilities for verification SOP class
ae.addTransferCapability(
new TransferCapability(null,
/* SOP Class */ UID.Verification,
/* Role */ TransferCapability.Role.SCU,
/* Transfer syntax */ TRANSFER_SYNTAX_CHAIN)
);
// Setup presentation context
rq.addPresentationContext(
new PresentationContext(
rq.getNumberOfPresentationContexts() * 2 + 1,
/* abstract syntax */ UID.Verification,
/* transfer syntax */ TRANSFER_SYNTAX_CHAIN
)
);
rq.addRoleSelection(new RoleSelection(UID.Verification, /* is SCU? */ true, /* is SCP? */ false));
try {
// 1) Open a connection to the SCP
Association association = ae.connect(local, remote, rq);
// 2) PING!
DimseRSP rsp = association.cecho();
rsp.next(); // Consume reply, which may fail
// Still here? Success!
// 3) Close the connection to the SCP
if (as.isReadyForDataTransfer()) {
as.waitForOutstandingRSP();
as.release();
}
} catch (Throwable ignore) {
// Failure
}
Another example, retrieving studies from a PACS given accession numbers; setting up the query and handling the result:
String modality = null; // e.g. "OT"
String accessionNumber = "1234567890";
//--------------------------------------------------------
// HERE follows setup of a query, using an Attributes object
//--------------------------------------------------------
Attributes query = new Attributes();
// Indicate character set
{
int tag = Tag.SpecificCharacterSet;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
query.setString(tag, vr, "ISO_IR 100");
}
// Study level query
{
int tag = Tag.QueryRetrieveLevel;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
query.setString(tag, vr, "STUDY");
}
// Accession number
{
int tag = Tag.AccessionNumber;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
query.setString(tag, vr, accessionNumber);
}
// Optionally filter on modality in study if 'modality' is provided,
// otherwise retrieve modality
{
int tag = Tag.ModalitiesInStudy;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
if (null != modality && modality.length() > 0) {
query.setString(tag, vr, modality);
} else {
query.setNull(tag, vr);
}
}
// We are interested in study instance UID
{
int tag = Tag.StudyInstanceUID;
VR vr = ElementDictionary.vrOf(tag, query.getPrivateCreator(tag));
query.setNull(tag, vr);
}
// Do the actual query, needing an AppliationEntity (ae),
// a local (local) and remote (remote) Connection, and
// an AAssociateRQ (rq) set up earlier.
try {
// 1) Open a connection to the SCP
Association as = ae.connect(local, remote, rq);
// 2) Query
int priority = 0x0002; // low for the sake of demo :)
as.cfind(UID.StudyRootQueryRetrieveInformationModelFind, priority, query, null,
new DimseRSPHandler(as.nextMessageID()) {
#Override
public void onDimseRSP(Association assoc, Attributes cmd,
Attributes response) {
super.onDimseRSP(assoc, cmd, response);
int status = cmd.getInt(Tag.Status, -1);
if (Status.isPending(status)) {
//--------------------------------------------------------
// HERE follows handling of the response, which
// is just another Attributes object
//--------------------------------------------------------
String studyInstanceUID = response.getString(Tag.StudyInstanceUID);
// etc...
}
}
});
// 3) Close the connection to the SCP
if (as.isReadyForDataTransfer()) {
as.waitForOutstandingRSP();
as.release();
}
}
catch (Exception e) {
// Failure
}
More on this at https://github.com/FrodeRanders/dicom-tools

Checking domain name age through Java

I am making a simple phishing scanner tool for a university project. One of my detection methods includes checking if the DNS within the email are valid, and I also want to check their age. This is example code of how I check if they are existing:
import javax.naming.NamingException;
import javax.naming.directory.Attribute;
import javax.naming.directory.Attributes;
import javax.naming.directory.DirContext;
import javax.naming.directory.InitialDirContext;
import java.util.Hashtable;
public class DNSExample {
static int doLookup( String hostName ) throws NamingException {
Hashtable env = new Hashtable();
env.put("java.naming.factory.initial",
"com.sun.jndi.dns.DnsContextFactory");
DirContext ictx = new InitialDirContext( env );
Attributes attrs =
ictx.getAttributes( hostName, new String[] { "MX" });
Attribute attr = attrs.get( "MX" );
if( attr == null ) return( 0 );
return( attr.size() );
}
public static void main( String args[] ) {
String [] array = {"google.com","dsad33114sssaxzx.com"} ;
for( int i = 0; i < array.length; i++ ) {
try {
System.out.println( array[i] + " has " +
doLookup( array[i] ) + " mail servers" );
}
catch( Exception e ) {
System.out.println(array[i] + " : " + e.getMessage());
}
}
}
}
How would I need to modify the above code to include a check of age
for servers that exist?
I think you've chosen a problem that cannot be solved in the general case ... using current generation internet standards:
The information you need cannot be obtained from DNS itself.
In some cases information about DNS registrations can be obtained from WHOIS. However, the information returned by WHOIS servers is not standardised:
There is no standard information model.
There is no standard format.
There are no guarantees as to the accuracy of the information.
It is not even clear if "age of server" is going to be available. (For instance, the closest that APNIC's WHOIS provides to that is the last modification timestamp for the DNS record. And that is NOT a good proxy for server age.)
There is a set of RFC's that define something called CRISP, but as far as I can make out the purpose of that standard is for registrar to registrar exchange of information. (I couldn't find any public-facing services based on CRISP.)
There is also an IETF working group called WEIRDS which I think is intended to define a web-enabled replacement for WHOIS. (Don't confuse WEIRDS with the IETF WEIRD WG!) But that was formed very recently, and it is too soon to make any predictions of the outcome. (Or how long it will take for the NICs to implement any specs that come out of the WG.)
Summary: your chances of implementing something in this space that really works are currently low. Probably the best you can hope to achieve is something based on screen-scraping one or two WHOIS services.
This might change in a few years, but that is of no help for your current project.
It seems based on your description and comments above you are trying to gather whois information.
download APIs from http://commons.apache.org/proper/commons-net/
change the nameToQuery below and run it.
public class WhoisIt {
public static final String WHOIS_SERVER = "whois.internic.net";
public static final int WHOIS_PORT = 43;
public static void main(String[] args) throws Exception {
String nameToQuery = "avajava.com";
WhoisClient whoisClient = new WhoisClient();
whoisClient.connect(WHOIS_SERVER, WHOIS_PORT);
String results = whoisClient.query(nameToQuery);
System.out.println(results);
}
}
good luck

log4j: Standard way to prevent repetitive log messages?

Our production application logs an error when it fails to establish a TCP/IP connection. Since it is constantly retrying the connection, it logs the same error message over and over. And similarly, other running components in the application can get into an error loop if some realtime resource is unavailable for a period of time.
Is there any standard approach to controlling the number of times the same error gets logged? (We are using log4j, so if there is any extension for log4j to handle this, it would be perfect.)
I just created a Java class that solves this exact problem using log4j. When I want to log a message, I just do something like this:
LogConsolidated.log(logger, Level.WARN, 5000, "File: " + f + " not found.", e);
Instead of:
logger.warn("File: " + f + " not found.", e);
Which makes it log a maximum of 1 time ever 5 seconds, and prints how many times it should have logged (e.g. |x53|). Obviously, you can make it so you don't have as many parameters, or pull the level out by doing log.warn or something, but this works for my use case.
import java.util.HashMap;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
public class LogConsolidated {
private static HashMap<String, TimeAndCount> lastLoggedTime = new HashMap<>();
/**
* Logs given <code>message</code> to given <code>logger</code> as long as:
* <ul>
* <li>A message (from same class and line number) has not already been logged within the past <code>timeBetweenLogs</code>.</li>
* <li>The given <code>level</code> is active for given <code>logger</code>.</li>
* </ul>
* Note: If messages are skipped, they are counted. When <code>timeBetweenLogs</code> has passed, and a repeat message is logged,
* the count will be displayed.
* #param logger Where to log.
* #param level Level to log.
* #param timeBetweenLogs Milliseconds to wait between similar log messages.
* #param message The actual message to log.
* #param t Can be null. Will log stack trace if not null.
*/
public static void log(Logger logger, Level level, long timeBetweenLogs, String message, Throwable t) {
if (logger.isEnabledFor(level)) {
String uniqueIdentifier = getFileAndLine();
TimeAndCount lastTimeAndCount = lastLoggedTime.get(uniqueIdentifier);
if (lastTimeAndCount != null) {
synchronized (lastTimeAndCount) {
long now = System.currentTimeMillis();
if (now - lastTimeAndCount.time < timeBetweenLogs) {
lastTimeAndCount.count++;
return;
} else {
log(logger, level, "|x" + lastTimeAndCount.count + "| " + message, t);
}
}
} else {
log(logger, level, message, t);
}
lastLoggedTime.put(uniqueIdentifier, new TimeAndCount());
}
}
private static String getFileAndLine() {
StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();
boolean enteredLogConsolidated = false;
for (StackTraceElement ste : stackTrace) {
if (ste.getClassName().equals(LogConsolidated.class.getName())) {
enteredLogConsolidated = true;
} else if (enteredLogConsolidated) {
// We have now file/line before entering LogConsolidated.
return ste.getFileName() + ":" + ste.getLineNumber();
}
}
return "?";
}
private static void log(Logger logger, Level level, String message, Throwable t) {
if (t == null) {
logger.log(level, message);
} else {
logger.log(level, message, t);
}
}
private static class TimeAndCount {
long time;
int count;
TimeAndCount() {
this.time = System.currentTimeMillis();
this.count = 0;
}
}
}
It would be fairly simple to control this by recording a timestamp each time you log the error, and then only logging it next time if a certain period has elapsed.
Ideally this would be a feature within log4j, but coding it within your app isn't too bad, and you could encapsulate it within a helper class to avoid boilerplate throughout your code.
Clearly, each repetitive log statement would need some kind of unique ID so that you could merge statements from the same source.

Categories