refactor data processing

[seanses]

Refactor data processing to

1. change clean API as a non-async-iterator buffer based API (can drop async next as we drop async in underlying crates), usage:

let pft = PointerFileTranslatorV3::new(config).await;

/* ----------- Clean file 1 (can safely spawn into another thread) ----------- */
let cleaner = pft.start_clean(4096 /*buffer size*/, Some(path1)).await?;
while let Some(data) =  read_file(&mut reader1) {
    cleaner.add_bytes(data).await?;
}
let cleaned_result = cleaner.result().await;

/* ----------- Clean file 2 (can safely spawn into another thread)  ----------- */
let cleaner = pft.start_clean(4096 /*buffer size*/, Some(path2)).await?;
while let Some(data) =  read_file(&mut reader2) {
    cleaner.add_bytes(data).await?;
}
let cleaned_result = cleaner.result().await;

/* ----------- Finish ----------- */
pft.finalize_cleaning().await

For example, see

xet-core/rust/gitxetcore/src/data/data_processing_v3.rs

Line 521 in 2c94529

let cleaner = self.start_clean(4096, Some(path)).await?;

2. drop XetConfig dependency. Right now there are some helper functions to map XetConfig to new configurations (see

   xet-core/rust/gitxetcore/src/data/configurations.rs

   Line 124 in a36954d

   pub async fn translator_config_from(

   ), these are just for testing the correctness of the new data processing logic using the existing test set up.

3. make repo salt optional for dedup

All integration tests pass.
Same clean speed as before.