Adityon » Data compression

Random data compression – Lowerband & upper band filtering

Keshav Shetty — Sun, 09 Jan 2011 15:12:25 +0000

Wish you all happy new year 2011.

in this article I will explain how a set of unique elements can be represented with less bits using filtering elements into two sets of upper and lower band.

Few users suggested to change the title, because title says “Random data” where as article describes about unique elements. Let me clarify that million random digit contains average 90-110 bytes of duplicate or non unique for every 256 bytes. If we could represent unique data set <=202 byte(worst case), we can use remaining 54 byte to represent duplicates. Refer previous article here

Coming to filtering technique, In every set of size 2^n, we will have exactly 2 set where half elements belongs to upper part and half elements belongs to lower part. e.g: In a set 256 unique values we can filter first or upper array containing element between 0 to 127 and lower array 128 to 255. By parsing and remembering using a flag(false for upper and true for lower set) weather element moved to upper or lower set we can locate the original position. Apply this method for newly created set recursively.

Lets take the same the input used in previous article of size 16, i.e: (7, 12, 9, 3, 13, 6, 8, 15, 1, 4, 2, 11, 0, 14, 5, 10)

Element 0-7 belongs to upper set and 8-15 belongs to lower set. Upper set or lower set will have equal number of elements of size (input set size)/2. When a element moves to upper set we can remember position as false and if element moves to lower set we can remember position as true. Once either upper or lower set size touches the size of the element in new set (i.e actually half of the original input size) we can stop parsing further.

In the above example – Total number of element=16, Size of new subset = 8, Elements in first/upper set is 0-7 and elements in lower set 8-15. After applying first time filter we get following output.

This method is similar to reverse merge sort, difference is here we use top to bottom approach, whereas merge sort uses bottom to up.

Using this method we can represent 256 unique elements and save 128 byte(best case) to 31.75 byte (worst case) as illustrated below.

Worst case happens when last element of each set belongs to other set and vice versa.

As I said beginning this method not sufficient to address million random digit, where I need to save at least 54 byte for every 256 byte.

In the next article I will write about using unidirectional graph how we can represent set of unique elements. Till then please pass your comments.

Random data compression – Risk based selection

Keshav Shetty — Sun, 03 Oct 2010 15:39:18 +0000

In my previous article I mentioned that once tree structure is ready we can fill and generate the random input along with one bit diff encoding.

As I observed as tree reaches mid levels, available options increases to 64+ elements, which will result into minimum 6bits per selection.

With respect to million random digit this approach will result into not more than 27 duplicates can be accommodated.

We need a better approach to selecting elements instead of one bit dif encoding.

I used the modified version of one bit dif encoding, I call it as risk based selection.

Lets assume that I have possible available selection (4, 9, 40, 102, 150, 165, 172, 197, 245) and actual selection was 245. Using one bit dif encoding I need 3 bit i.e “111″ to select 245 among these input.

In risk based selection I will try to make two list with maximum numbers in one list and remaining in another list based on particular bit.

If you analyze below image you will understand that 4th bit in available option results into 7 zeros, If I knew 4th bit of number which I am selecting is “1″ then I will left with only two remaining numbers.

i.e in available number 4th bit has maximum variance. (As I am aware there is other 6th bit as 1 as maximum or 7th bit as maximum zero, lets select first possibility)

The number which I am going to select i.e 245 has 4th bit as “1″, If I store this bit my next available options reduces to two i.e 150, 245.

As you can see in the image red marked bit are selected for next round. I repeat same procedure for remaining options until I am left with only one option.

Green box indicates 2nd level filtering. After second bit selection I will left with 245 which is the number I am looking for. This way I need only two bit (i.e “11″) to select particular node.

This selection is more effective than bit dif encoding which requires 3 bit compared to 2 bit.

Note: In worst case scenario we may end up with 8bits for each node. You can see yourself by applying this for below example

Assume we have these available option (1,2,4,8,16,32,64,128) and number to be selected is 1. We will end up with 8 bits.

Random data compression – Using Tree

Keshav Shetty — Sun, 03 Oct 2010 11:14:12 +0000

In this article I will explain how a B Tree can be used to model the unique numbers. This will be interesting article as best case scenario requires only 64bytes.

As you are aware B Tree or a balanced tree contains maximum two child for each node, all nodes left side will be lower than current node and all right side nodes will be higher than current node.

How can we use B Tree to model the random unique numbers?

In case of 256 unique numbers we will have 256 nodes, left most node starts with zero and right most node 255. For each node we need two flags to identify weather it contains left node and right node.

Recursively we can build the tree structure. Once tree structure is ready we can fill each node with value by using pre-order traverse.

Lets take an example, Assume we have random 16 inputs (7, 12, 9, 3, 13, 6, 8, 15, 1, 4, 2, 11, 0, 14, 5, 10)

Respective tree will be

If we knew tree structure we can refill values of each node by starting from left most to rightmost.

However after building tree we need addition info find which child node appears next.

In the above example we are sure that 7 appeared first in the input array. Next number either 3 or 12 (one of the open child).

We need one bit to select either 3 or 12(That bit will be “1″). we end up with 12 as next input. Now as next input should be within 3, 9, 13. We can go on applying one bit dif encoding to select proper child.

Whenever a child is selected its children are added to available options for selecting next node.

This way we need 256bits(for node flag) + one bit dif encoding info of each child.

Best case and worst case scenario

Using tree in best case scenario i.e each node has maximum only one node, that happens when input is in ascending order we need just 2^(n+1) bits i.e for 256 input we need 512bits or 64 bytes. JUST 64 byte for 256 unique values!!!.

In worst case scenario we need 64byte + bit diff info of worst case = 2047bits (i.e ~256 bytes) No saving.

Worst case scenario happens when child node added to available options is consumed at the end.

Using this technique by rearranging million random digit I could compress up to 27bytes non unique numbers. Worst than merge sort. This technique will be effective provided we find a way to increase the level of tree which results into nodes with single child more. I will write more about this later.

Random data compression – diff encoding with merge sort

Keshav Shetty — Sat, 02 Oct 2010 18:12:31 +0000

Last article didn’t go well with my readers, so I decided to elaborate with example.

Lets assume we have eight random input say 5, 1, 3, 1, 2, 3, 0, 5

In this input 4th, 6th and 8th element are non unique i.e they already appeared at least once.

For each number we need a identifier or a flag to identify it as unique or duplicate. So we will have bit info like 0 0 0 1 0 1 0 1

When 4th element appears which is duplicate of one of the past unique, that should exist within (5, 1, 3)

Similarly 6th element should exist within (5, 1, 3, 2) and same applies to 8th element which should exist within (5, 1, 3, 2, 0)

Lets go back to the algorithm mentioned in my previous article.

1. Read 256 bytes from the input (In the above example we will read 8 input of each 3 bit)

2. Sequential process and mark unique and duplicates, that results into 0 0 0 1 0 1 0 1. We need to store this info to be used at the time of decompression. For above 8 input we need 8bits (for 256 input we need 256bits).

3. For each marked as non unique data, remember the position using one bit diff encoding.

i.e for 4th element which is duplicate of 2nd element, using one bit diff encoding we get “01″ and for for 6th element we get “10″ for 8th element we get “000″. The complete table of one bit diff encoding available here

4. Remove non unique data and rearrange the list by moving forward, at the end empty space fill with missing numbers. Refer above image bottom array is rearranged one.

Now the array contains only unique elements, we can apply reverse merge sort to sort and store these bit info.

So compressed file contains unique or duplicate flags generated in step(2) + one bit diff encoding info of duplicate numbers generated in above step(3) + merge sort bit info.

While decompressing use merge sort bit info to re create unique array and use unique or duplicate flags to note which was duplicate and then use bit dif info refill duplicates.

I hope this example clears many of your doubts.

Next article will be interesting one, which is based on B Tree.

Random data compression – One bit diff encoding continued

Keshav Shetty — Fri, 01 Oct 2010 12:42:14 +0000

This is continuation of my previous article on Random data compression-One bit dif encoding.

Sorry for the long gap between this and last article. I was too busy.

After my my previous article many had asked how I achieved compression up to 42 bytes duplicates.

Here we go with the algorithm I used.

1. Read 256 bytes from the input.

2. Sequential process and mark unique and duplicates, we need 256 bits or 32 bytes. (Can’t stop this loss)

3. For each marked as non unique data, remember the position using one bit diff encoding. (Details in previous article) Requires much lesser bits than 8bits depending on non unique data appearance. We will consume lesser bits if the non unique data appears at the beginning rather than at the end of the list.

4. Remove non unique data and rearrange the list by moving forward, at the end empty space fill with missing numbers.

5. Apply merge sort and store bit information of sorting data. (Refer how to use merge sort article)

While decompressing

1. Use merge sort data to recreate the unique data list.

2. Apply non unique flags to rearrange the list.

3. Use one bit dif encoding to fill duplicate or non unique numbers.

Using above technique I could compress random data up to 42 bytes. I repeat up to, not always. I mean if the non unique appears at the beginning of the list we can accommodate up to 42 bytes, whereas if the non unique appears at the end of the list we can hardly compress data up to 12~14bytes.

Note: This method is not effective for Million Random Digit from RAND. We need different approach.

Random data compression – One bit diff encoding

Keshav Shetty — Sat, 05 Jun 2010 18:10:03 +0000

This is continuation of my previous article on Random data compression-How to use merge sort?.

As I mentioned earlier – it is hard or impossible(as of now) to compress million random digit. As per my analysis million random digit file contains around 90 – 110 bytes duplicates or repeated number within every block of 256 bytes. When input data is very pure or near pure(I mean uniqueness) or when input data is highly polluted or noise(I mean duplicates), then it is easy to compress. May be we can borrow the idea from chemistry. i.e if you refer periodic table lighter elements can be combined using nuclear fusion(Stars) or higher elements can be broken using nuclear fission(Uranium). In both cases energy released which we can relate to compression, however if you refer periodic table you will understand that elements with highly stable nucleus hard to break or fuse. e.g: Lead.

Same pattern I can see in million random digit where if uniqueness reaches extreme high or extreme low we can easily compress the data.

My so far developed compressor could achieve compression with doubles less than 42 or more than 156 in every 256 bytes.

Before I go further about my compression theory, I have to write one more article about one bit difference variable encoder. This is one of the known algorithm based on probability or possibilities.

About bijective you might have observed it is one on one mapping, however this fails to address relation within elements of donor domain set or acceptor domain set.

The one bit variant encoding based on the possibilities. i.e if we have 4 unique possibilities then we need 2 bits (00,01,10,11) to represent all possible values. How many bits do we require when we have 5 possibilities or say 6 possibilities?

In one of my earlier article I mentioned usage of insertion sort or bubble sort stating using this technique we can represent for every possible combination we will have 1792 bits or 224bytes. But I need to re factor this statement. We need much lesser bits.

Assume that you have a sorted list of 256 all unique values, when a random input comes lets say 26, you pick from the initialized list, actually you pick the position, Since it is first input and there are 256 possibilities we need eight bit to represent the position. For next random input we need to chose among remaining 255 possibilities, in this case exception for last position we need eight bit. Here it depends on the position which we pick from remaining elements. We can calculate required bits for best case scenario(when input are in sorted order) and worst case(when input are in reverse sorted order) as below.

If you refer the below illustration you will quickly understand that as possibilities changes required bits changes by maintaining uniqueness of bit stream.

The cells in yellow color are one bit diff encode based first entry in each row.

Using above calculation you can represent using 1545-1793bits(193.125-224.125bytes) for every 256 unique bytes.

Next article I will explain how I achieved compression for 42doubles.

Random data compression – Is it possible? How to use merge sort?

Keshav Shetty — Thu, 03 Jun 2010 14:14:45 +0000

This is continuation of my previous article on Random data compression possibilities.

Some of the readers asked how reverse merge sort (merge unsort) can be used to represent 256 unique values using 128bytes(best case) or 224.25(worst case).

As illustrated in previous article using merge sort we sort the random input and store bit information of the list from where we picked the smaller number. Actually we are reshuffling the original position and stored bit information represents how a input changed its position from original list to sorted list. Lets take an example of a random array of 16 numbers varying from 0…15.

As shown in above image after sorting, the position are reshuffled and stored bit information exactly represent from where the each element moved.

The encoding code is recursive function, to start applying merge from two list of size 2 to merge the two list of 128. You can refer the mergesort in action here.

The merge sort encoding code given below(C++).

   void Transform::mergeSort(unsigned int beginPosition, unsigned int endPosition) {
                unsigned int midPosition = beginPosition + (endPosition-beginPosition)/2;
                if (midPosition>beginPosition && midPosition0 && pendingInList2>0) {
                        if (list1[list1Pointer]0) {
                        while(pendingInList1>0) {
                                transformedByte[mainArrayPointer] = list1[list1Pointer];
                                mainArrayPointer++;
                                list1Pointer++;
                                pendingInList1--;
                        }
                } else {
                        while(pendingInList2>0) {
                                transformedByte[mainArrayPointer] = list2[list2Pointer];
                                mainArrayPointer++;
                                list2Pointer++;
                                pendingInList2--;
                        }
           }
        }

As you observed above when items in one list gets over (either first list or second list) for remaining element we don’t require sorting bit information.

Please note in above code random input is stored in transformedByte variable & initilized in separate method for each 256byte input. The object bitWriter is a separate class of type BitWriter(Utility class) which is initialized in constructor. BitWriter will accumulates bit information and writes to output file when accumulated bits crosses 1 byte.

The merge sort decoding code given below(C++).


  void Transform::reverseMergeSort(unsigned int beginPosition, unsigned int endPosition) {
        unsigned int midPosition = beginPosition + (endPosition-beginPosition)/2;
        if (midPosition>beginPosition && midPosition0 && pendingInList2>0) {
                bool nextBit = bitReader.genNextBit(); // Read bit information from sorted encoding
                if (nextBit==false) {
                        inputPositionArray[mainArrayPointer] = list1[list1Pointer];
                        mainArrayPointer++;
                        list1Pointer++;
                        pendingInList1--;
                } else {
                        inputPositionArray[mainArrayPointer] = list2[list2Pointer];
                        mainArrayPointer++;
                        list2Pointer++;
                        pendingInList2--;
                }
        }
        if (pendingInList1>0) {
                while(pendingInList1>0) {
                        inputPositionArray[mainArrayPointer] = list1[list1Pointer];
                        mainArrayPointer++;
                        list1Pointer++;
                        pendingInList1--;
                }
        } else {
                while(pendingInList2>0) {
                        inputPositionArray[mainArrayPointer] = list2[list2Pointer];
                        mainArrayPointer++;
                        list2Pointer++;
                        pendingInList2--;
                }
        }
}

Before calling reverseMergeSort inputPositionArray is initialized with 0,1,2.. as default initial position. The object bitReader is a separate class of type BitReader(Utility class) which is initialized in constructor and provides API to read a bit at a time(not byte), which internally reads a byte and gives a bit at a time. These utility classes are developed by me exclusively for above purpose.

After exiting reverseMergeSort the inputPositionArray does not represent original input, instead it represent the positions of input in original data. After exiting we have to go thru one more loop to construct original input as below.

   for(int i=0; i<256;i++) {
       inputData[inputPositionArray[i]] = i;
   }

Now inputData contains original data.

I will inform in next article how I compressed random data in which we have maximum 42 duplicates(non unique) for every 256 input byte.

Random data compression – Is it possible? (Part 2)

Keshav Shetty — Sun, 20 Dec 2009 10:44:15 +0000

This is continuation of my previous article on Random data compression possibilities.

Compression of unique values

As I indicated in previous article if for every 256 bytes each value appears only ones, then we can achieve the compression using any of the below techniques.

1. Using insertion sort – by remembering the position.
Using this technique for every 256 byte – we can save exactly 32 bytes, i.e for first element no need to remember any position, for next value one bit sufficient to remember weather current element inserted after or before previous number, for next two numbers we need two bits for each value, for next 4 numbers we need 3 bits for each number and so on until for last 128 values we need 8 bit for each value. It is constant save – i.e for every possible combination we will have 1792 bits or 224bytes. (We save 256 – 224 = 32bytes)

2. Using factorial of 256
In this approach, all possible variations can represented using factorial of 256, that will end up with 1684bits for every possible combination. It is constant save – i.e for every possible combination we will have 1684 bits or 210.5 bytes. (We save 256 – 210.5 = 45.5bytes)

3. Using reverse merge sort – Using this approach use a merge sort and store bit information of the list from where we picked the smaller number. Advantage of this technique is, when one list gets over, we don’t require to store or remember unsorting information for remaining element in other list, we just need to append it as illustrated below.

In best case scenario(when all elements are in ascending order) we just need (1/2*8bit) x 256 = 128 bytes for total 256 input bytes, in worst case we need 1793 bits as illustrated below.

Depending on the unique numbers appear in the list we can save between 128 byte(best case) to 31.75 byte (worst case)

Similarly we can use reverse quick sort technique, however merge sort is better than quick in storing minimal unsorting information, because in merge when one list gets over, we don’t require unsorting information for remaining elements in other list, this feature missing if we use quick sort, but still reverse quick sort can be used with lesser than 2048bits.

How to use above theory when all numbers are not unique?

Well above said theory works when all numbers are unique, but in real time such scenario is rare, instead there will be some duplicates and some unique numbers missing.

In fact random digits created by the RAND group contains on an average 100bytes duplicates(repeated) in every block of 256 numbers. in such data we need to patch the dataset by removing non unique, and inserting missing unique numbers, so that above reverse merge sort can be applied. When we remove duplicates we need to retain its position and actual value. (No need to retain the introduced missing numbers, which will be discarded after unsorting. But to patch this way we need lot of byte, e.g: if 100 numbers is duplicates (i.e 100 numbers are missing), we need to have 100*2=200 bytes. (very costly).

However there is better patch as below
1. Read 256 block of data, mark the duplicates/non unique, remember these relative to current position of the number (so we don’t require 8bits for position), Now we have missing unique numbers.
2. Rearrange the input data by eliminating duplicates so that unique numbers moves upper part of the list, fill missing unique numbers at the end of list. (make sure all missing numbers are filled in ascending order so that we can avoid storing unsorting information of missing numbers – which is of nu use for us – This way we can save few bytes)
3. Perform merge sort on the list excluding introduced missing numbers, now the compressed file will have Unsorting information + duplicate number positions + duplicate values.
4. If missing numbers are less than 32 byte we can actually store position using eight bits per position, if it is more than 32, then we use a bit for each byte indicating duplicate or not.

Using above technique I developed a compressor which can compress random data up to 42 duplicates or missing unique numbers. (I used million digit of Rand with some modification to maintain maximum 42 missing numbers in every 256 input bytes. However I am yet to find way to address another 60 bytes so that million digit of Rand group can be compressed at least by a byte.

I will fine tune my compressor and announce in next article in next year. (After Xmas/new year), next article I will introduce another technique of transition or state representation for random data.

Wish you all happy new year 2010.

Random data compression – Is it possible? Part 1

Keshav Shetty — Mon, 07 Dec 2009 13:55:53 +0000

Random data compression lossless – Is it possible?

The answer is NO (…. and yes!!!)

This is little lengthy blog to show and prove the difficulties and possible ways. This article divided into three section.

(Before you read further I suggest you to read Mark Nelson “The Million Random Digit Challenge“)

1. Why it is not possible to compress random data. (brief introduction to Kolmogorov theory)

2. Why it is possible (Quantum theory, matter and anti matter introduction)

3. The future for compression and possible solution with transition representation and unsorting techniques

If you want to skip this introduction and continue with next article, please click – using unsorting techniques for random data compression.

1. Why it is not possible to compress random data. (brief introduction to Kolmogorov theory)

As per pigeonhole principle – if n pigeons are put into m pigeonholes with n > m, then at least one pigeonhole must contain more than one pigeon or if n

The Russian mathematician Kolmogorov proved thru his complexity theory that – With the uniform probability distribution on the space of bitstrings of length n, the probability that a string is incompressible by c is at least 1 − 2^(-c+1) + 2^−n (More details please refer to wikipedia here)

Lets take a arithmetic approach to prove the above theory.

The byte i.e 8 bit can represent 256 variety of data i.e 0, 1, 2, ….255 – As you can see there are totally 256 variation or possible values and assume if each one appears only once(in 256 byte) then you cannot represent all these possibilities with a domain contains smaller set than 256.

All existing compression algorithms can compress the data if a data set where all possibilities are not present. e.g: In a input data if only 8 unique bytes present, then we just need 3 bits(0, 1, 2, ….7 – total possibilities) to represent each byte of the input data.

Note: Each algorithm works differently, however all existing algorithms like dictionary based, LZW, Huffman, RLE etc works based on patterns, repetitions, appearance, occurrence or uniqueness of the data.

People often fail to digest the simplicity of the above theory and often claim they have developed magic data compressor which can compress any type of data. (Refer comp.compression)

(Note: I am not claiming any such magic compressor – but I claim such possibilities)

2. Why do I believe that random data compression is possible?

Don’t mistake me, I strongly believe in Kolmogorov complexity and number theory, It is impossible to develop magic compressor which can compress any type of DATA.

In my school days I learnt that our universe was in the form of bindurupi (in the form of dot or originated from nothing) and expanded (still expanding), generating so many possibilities and varieties. As per the quantum theory matter and anti matter joins together – during this process new variations appears. Better example is each couple produce totally different baby, the new baby is neither 100% of father or 100% of mother (or not even 50% father + 50% mother), Baby might be having few feature of father and mother and ancestors, but it also got its own uniqueness. Genes keep mutating, but based on what?

Note: “Nothing” doesn’t mean empty or zero, we are still not clear about nothing. In Sanskrit and Sanathan mythology it is mentioned about Brahma vidya which deals with separation of matter and anti matter.

A matter is a materialistic world or physical world which we can see, we can feel or we can represent, and store (Random data is also a type of matter), whereas as anti matter we cannot imagine or can be represented, even quantum theory couldn’t provide any visual way to see or feel this. (May be human beings are not yet ready to digest this because we are still living in materialistic world)

As per Einstein’s Energy mass relationship E = mc^2 which states “E” energy generated when a object of mass “m” is converted to energy, neither mass nor energy were conserved separately. This is basic used in Nuclear fusion.

Now you might be thinking – What is the relation between Einstein’s mass/energy theory and random data compression?

In one of the lecture Einstein mentioned that you can convert mass “m” to energy “E”, neither mass or energy is destroyable. Similarly you can convert energy to matter !

One of the audience asked Einstein how is it possible to generate mass from energy? – effectively if you have enough energy you can generate any type of object like Gold, Uranium etc.

Einstein said it is outside the definition of Physics and a new world opens thru spiritual way to prove it. Mostly Einstein had a idea of anti matter at that time. In my view anti matter is huge energy reservoir which acted on matter to generate variety of materials we see today. (Imagine Nuclear fission of heavier material), please note I believe there should different type of antimatter similar to matter we see)

We heard many such information in ancient India and even current time it is believed that few got such capability to generate objects. e.g:

1. Pandavas had Akshya patre – to generate food
2. Kamadhenu – A holy cow could provide anything sage Vasista asked for
3. Devine Sri Sathya Sai Baba

I will not go further in spiritual way, which I cannot address properly or I have no sufficient knowledge in this area.

As long as we treat the material as data we cannot compress the random data, the day we find the way to represent the anti matter we can achieve this or in other words we should deal with existing random data as transition or state representation, because data is static form of Bhrahma vidya, it is anti matter which generates all variations.

What does it mean transition or state representation?

There exist a huge set of static data – The size of static data is beyond the imagination of human being. The so called anti matter acts on these matter and generates new set of static data. (This is how universe is expanding?)

The static data is growing! then how come it is static? (I used static to represent default or initial set) During this process there is a different stage of the matter it passes thru. e.g: Radio activity material decays over a period of long time to generate another material, during this long process in the intervals we have variety of data or material, if we know the state of the item we can accurately tell the process involved in it or time involved in it, Similarly if we knew all the process it passed thru, then we can define the state of the material.

Kolmogorov turing machine is actually generating data from above said static data. We call it as highly random or high entropy because we cannot define it as set or within domain. We human beings not reached the stage where we can define the random. Note: Random doesn’t mean uncertainty or non guessable future, but it is predefined beyond the imagination or understanding limit of materialistic human.

Summary : As long as we treat the random numbers as data we cannot compress it, because number theory which applicable to data clearly proves that it is impossible. However the day if we find the way to treat them as either anti matter or state transition then we can change the way we represent the data.

3. The future for compression and possible solution with transition representation and unsorting techniques

In above section I mentioned about state or transition representation, now if we represent random data as state or transition then we will see totally different scenario or concept.

I will take a small portion of the static data I mentioned above, i.e a subset of 256 unique values. That mean I need 8 bits to represent all variations. If in every 256 data each value appears only once then none of the existing algorithm can compress the data. (Although such entropy chances in real time is not high, but they are highly random that cannot be compressed)

[I will try to prove this using conventional manner instead of mathematical form, I don't want my readers get confused with mathematical formulas, I will try to avoid any use of formula or function unless it is un-avoidable].

Using a unsorting algorithm we can achieve compression of such data between 10% to 50%!!!.

The catch here is data set is fixed, but only their positions are different or data not organized. When all numbers are unique we need to find a way to represent original position (Which requires lesser than 8 bits per value)

I hope I have not confused you, let me describe in other way

I have a set of 256 variations and all are unique – In this scenario I need 8bits to represent each value, The values might be 250, 140, 0, 130, 239, … and so on, each value appears only once.

If you use any of the existing compression algorithm we cannot achieve any compression, since there is no repetition or patterns present (Highly random and each value is unique)

In fact most of the compressor generates compressed data in the form of above mentioned form i.e highly unique and non repeatable patterns (Close to unique within every 256 block, but not exact) A better example is random digits created by the RAND group. Please refer Mark Nelson The Million Random Digit Challenge Revisited. (This data is very close to each 256 variations, but not 100% unique – that is based on Kolmogorov complexity)

But my methods can compress such random data (Not Kolmogorov complexity or his Turing machine based random data – because it is not 100% unique within 256 block, I am yet to reach there)

What is unsorting algorithm? In our graduation or engineering we studied various sorting algorithms like bubble sort, quick sort, merge sort etc, However we never came across anything called unsorting algorithm! – Why we need it?

Actually an unsorting algorithm is shuffling back the sorted items back into original order or position. Although such algorithms may not be that useful in real time applications, but I will demonstrate compression of 256 unique data using that.

If there are 256 data, then there are 256 positions that means still I need 8 bit to store each position. However unsorting algorithm is not just about placing the sorted order into original list, it also uses minimal place or storage required to represent original places.

Lets take merge sort, if we have list of two items, then we just need 1 bit(both best and worst case) to unsort back to original list, that bit represent which is smaller among two. If we have four items then we need 4 bits best case scenario and 5 bits in worst case, If we have 8 items then we need 12 bits best case and 17 bits worst case scenario. (Note: In case of 8 items we need actually 24 bits to represent all unique values), If we have 256 unique items we need 1024bits(128byte) best case and 1856 worst case. Actually 256 unique value requires 2048 bits. As you might observed using reverse merge sort we can achieve the compression mentioned above. I will not go in details, because it will fill another page. If you need further details please contact me. Please write me if you need a working code of such compression using reverse merge sort. Similar results can be obtained using reverse quick sort.

Above compressor works well for each value uniqueness within every 256 values, However in real time data is not that unique and Kolmogorov doesn’t guarantee that entropy within predefined set of data.

The above concept is same as probability theory, i.e If we toss a coin chances of getting either head or tail is 50%, However it won’t guarantee that for every two successive toss one will head and other will be tail. But over large number of toss you will end up with 50% head and 50% tail.

Coming back to random digits created by the RAND group – I observed for every 256 block of data uniqueness of values lies anywhere between 166 to 146, that means if we find a way to patch the missing unique number within every 256 block, then we reached the stage of random data compression, however patched unique number + unsorting information should be less than 2047 bits for every 256 block. I am still looking for such possibility.

Well now other approach, as I mentioned in previous section instead of representing random data as data, but as state or transition then new concept opens. i.e treat random digits of RAND group as unique transition of another process.

Continue reading next article – using unsorting techniques for random data compression.