Beta branch refactoring

[Tiihott]

Contains:

• Refactoring of Offset interface to KafkaRecord interface, with ability to print itself in SyslogRecord format.
• Implemented a separate KafkaRecordConverter class for converting the kafka records to SyslogRecord format.
• Refactoring of DataBaseOutput class to BatchDistribution (interface+class), which now allocates the consumed batch records to PartitionFile objects based on the record's topic partition.
• Implemented PartitionFile interface and class for storing the consumed batch records to files that are then stored to HDFS.
• Refactoring of WriteableQueue and Config classes to be immutable.
• Implemented consumer group rebalance handling using IngestionRebalanceListener class.
• Refactored tests and added new tests.
• Refactored maximum file size checks to be done after a record is added to the file, instead of checking the record size and file size before adding the record to the file.

Also contains after changes:

• Timeout handling if records are not coming in at a sufficient pace.
• Config refactoring.

[51-code]

Here are some general notes:
Multiple objects have long constructors with code in them, maybe that could be refactored?

Noticed that the useMockKafkaConsumer is a bit odd. Multiple objects have an if statement for checking this, could the objects instead implement an interface and provide a MockKafka version of the object?

Check which objects could be made final (public final class), probably most of them.

[Tiihott]

Pair session result: Minor naming change to improve naming conventions.

[kortemik]

perhaps we should drop the joke about Fuura and have a proper message now?

[Tiihott]

Solved in commit 7a21cda

[kortemik]

could you please elaborate this? try-with-resources to create a new syslogAvroWriter and then a close for it so it gets a double close. other thing is that no code in constructor

[Tiihott]

After some further testing and rechecking the logic how the syslogFile is handled, I deemed that initializing the avro-file in the constructor is redundant as the commitRecords() method will initialize the file if it doesn't exist yet.
Solved in commit 0a31fdd

[kortemik]

use cnf_01 for configuration

[Tiihott]

Implemented cnf_01 usage on commit 37d2a70

[kortemik]

validators are not necessary, use rather configuration in the methods like

public int timeout() {
String timeoutString = map.get("timeout");
if (timeout == null) {
throw new ConfigurationException("timeout not configured");
}

int timeout;
try {
timeout = Integer.valueOf()
}
catch (FormatException e) {
LOGGER.error("not a number");
throw e
}

if (timeout <= 0) {
 ...
}

return timeout
}

[Tiihott]

Implemented refactored configurations to the main code on commit 37d2a70

[kortemik]

use immutable partitionFileFactory or so, so the object created is no longer configurable but gets always immutable configuration and only the recordOffset provided to the partitionFileFactory.partitionFor(recordOffset). the partitionFileFactory must be either initialized in secondary constructor or be passed from the object creating this one directly to the constructor.

[kortemik]

apply similar pattern to the other at the moment configurable objects too, because objects must not be configurable.

[Tiihott]

Implemented PartitionFileFactory in commit 12bc003
Added improvements for it in commit 062ddcc and b6fe5dc

[Tiihott]

Infrastructure side changes are ongoing for Kafka, which will require changes to the project once they are complete.