How to exclude directories?

[ChristianBeer]

I want to scan a directory and all it's subdirectories excluding one substructure where I have constantly changing content which I don't care about at the moment.

The documentation of the -e flag is a bit sparse. I already tried to specify the directory but using strace I saw that duc indexed files in one of the directories. What kind of syntax does the -e flag understand?

Here is what I'm doing:
I have a /etc/ducrc file:

[global]
database /var/cache/duc.db

[ls]
recursive
classify
color

[index]
check-hard-links
one-file-system

I'm using duc-nox version 1.4.2-1 on Debian. The commands I already tried:

• duc index --exclude=/srv/some/structure/exclude /srv
• duc index --exclude=/srv/some/structure/exclude/*/* /srv the exclude directory has one level of subdirectories in which the actual files are

The general problem I have with these directories is that scanning them does not finish because files are continuously added and deleted and duc does not stop scanning the directories because there is always some new content in them. That is at least my assumption after letting it run for several days without it ever finishing.

[zevv]

Quoting Christian Beer (2017-01-30 16:50:46)

I want to scan a directory and all it's subdirectories excluding one substructure where I have constantly changing content which I don't care about at the moment. The documentation of the `-e` flag is a bit sparse. I already tried to specify the directory but using strace I saw that duc indexed files in one of the directories. What kind of syntax does the `-e` flag understand?

I'm surprised to see how long it took for this question to come up. The implementation of the flag is also a bit sparse. The `-e` option matches only relative path names, i.e. names without slashes in it. For example, there is no proper way to exclude /usr/local/bin. You could index using `-e bin`, but that would also exclude /bin and /usr/bin. This is a known problem, but up to now it did not seem to affect many people. The reason for this restriction that Duc does not keep track of complete path names during the directory traversal: it simply chdir()s its way through the file system, and only handles relative path names during indexing. At this time there is no need for Duc to keep track of /absolute/path/names, and adding this would add a overhead (albeit pretty small). I definitely can see the benefit of being able to properly specify paths to include or exclude, so maybe it is time to add this feature.

- `duc index --exclude=/srv/some/structure/exclude /srv` - `duc index --exclude=/srv/some/structure/exclude/*/* /srv` the `exclude` directory has one level of subdirectories in which the actual files are

What would probably work for you is using the option `--exclude=exclude`. Note that this will skip *any* path in your filesystem with the name `exclude`, so beware of side effects.

The general problem I have with these directories is that scanning them does not finish because files are continuously added and deleted and duc does not stop scanning the directories because there is always some new content in them. That is at least my assumption after letting it run for several days without it ever finishing.

That's kind of unexpected: Duc scans each directory only once, and will traverse the tree depth-first. If files are added after the initial readdir() for a directory, Duc should not notice. Do you see similar behaviour if you run the unix `find` command on the seem tree?

…

-- :wq ^X^Cy^K^X^C^C^C^C

[ChristianBeer]

Thanks for the quick answer.

What would probably work for you is using the option --exclude=exclude. Note that this will skip any path in your filesystem with the name exclude, so beware of side effects.

This could be working, but what about partial matches? I know that I have directories that are named exclude_1, exclude_2 and there could be some files that are named exclude.txt. The directories should also not be scanned because of one-file-system as they are symlinks to external NFS shares which I hope counts as a different filesystem for Duc.

That's kind of unexpected: Duc scans each directory only once, and will traverse the tree depth-first. If files are added after the initial readdir() for a directory, Duc should not notice. Do you see similar behaviour if you run the unix find command on the seem tree?

I usually don't run find on this tree as I already know it takes a lot of time because of the amount of files in the subdirectories (I can't even run ls -l on a single subdirectory). That's why I want to exclude this directory structure. I recently ran a du -d 1 on the structure and it finished within a day or so. Duc ran for something over 5 days (with the same IO throttling) before I killed it.

[zevv]

Quoting Christian Beer (2017-01-30 17:56:27)

This could be working, but what about partial matches? I know that I have directories that are named `exclude_1`, `exclude_2` and there could be some files that are named `exclude.txt`.

Yes, you can use wildcards here, so -e 'exclude_*' should work for you. I guess the documentation can be improved here.

should also not be scanned because of `one-file-system` as they are symlinks to external NFS shares which I hope counts as a different filesystem for Duc.

Symlinks are never followed, so that should work just fine.

I usually don't run `find` on this tree as I already know it takes a lot of time because of the amount of files in the subdirectories (I can't even run `ls -l` on a single subdirectory).

If you do, what happens? Does 'ls' go out of memory because of the sorting involved? How many files are in a directory here? I've tested Duc with 10M files in a single directory without problems, so it would be interesting to know what scale you're working on.

…

-- :wq ^X^Cy^K^X^C^C^C^C

[ChristianBeer]

I usually don't run find on this tree as I already know it takes a lot of time because of the amount of files in the subdirectories (I can't even run ls -l on a single subdirectory).

If you do, what happens? Does 'ls' go out of memory because of the sorting involved? How many files are in a directory here? I've tested Duc with 10M files in a single directory without problems, so it would
be interesting to know what scale you're working on.

It usually runs for a long time and uses all of the IO. The filesystem (xfs) and RAID setup in use here are mainly to blame. We experimented with ZFS and an optimized RAID setup which has a much better performance than this system I'm currently testing duc on.

The structure I'm excluding currently only has around 1K files per directory but there are 1024 subdirectories. I have ~400 other directories (that are scanned right now) which have around 750k files each. The process has accumulated 1h42m of CPU time and is constantly in D state waiting for IO to finish. The overall runtime is close to 20h and I'm throttling the duc process by sending it a STOP/CONT signal every 5 seconds (so it does not hog all the IO bandwidth) which mans actual runtime is around 10h. The partition I'm scanning is reported by df to be 3.8T with lots of very small files. So I don't know what runtime to expect.

The exclusion I now use seems to work. If you could improve the documentation on what is matched against the given string that would help a lot. I'll let my process run and monitor the current directory it is working on using ls -l /proc/$PID/fd/9 which always points to the currently scanned directory.

[zevv]

Quoting Christian Beer (2017-01-31 12:34:30)

The structure I'm excluding currently only has around 1K files per directory but there are 1024 subdirectories. I have ~400 other directories (that are scanned right now) which have around 750k files each. The process has accumulated 1h42m of CPU time and is constantly in D state waiting for IO to finish. The overall runtime is close to 20h and I'm throttling the duc process by sending it a STOP/CONT signal every 5 seconds (so it does not hog all the IO bandwidth) which mans actual runtime is around 10h.

In the past I have considered adding some tricks to Duc to avoid I/O hogging. My first idea was to have Duc do the appropriate ioprio_set() system calls to set the I/O scheduling class and prio, but the same can be done simply using `ionice -c3` to run Duc. Another very simple trick would be to add an optional short `usleep()` after reading each directory to leave some time for the rest of the system. I'll create a separate Github issue for this, as it is probably a feature worth adding.

The exclusion I now use seems to work. If you could improve the documentation on what is matched against the given string that would help a lot.

Agreed.

I'll let my process run and monitor the current directory it is working on using `ls -l /proc/$PID/fd/9` which always points to the currently scanned directory.

Thanks for that.

…

-- :wq ^X^Cy^K^X^C^C^C^C

[knedlsepp]

This also caused me quite a bit of pain. I've got some servers mounted in ~/mnt/ using sshfs...

[zevv]

Quoting Josef Kemetmüller (2017-04-04 16:04:41)

This also caused me quite a bit of pain. I've got some servers mounted in ~/mnt/ using `sshfs`...

Try

…

--fs-exclude=fuse.sshfs (check the exact file system type in /proc/mounts)

-- :wq ^X^Cy^K^X^C^C^C^C

[l8gravely]

>>>> "Ico" == Ico Doornekamp ***@***.***> writes:

Ico> Quoting Josef Kemetmüller (2017-04-04 16:04:41)

> This also caused me quite a bit of pain. I've got some servers mounted in ~/

Ico> mnt/ using `sshfs`... Ico> Try Ico> --fs-exclude=fuse.sshfs Hmmm... should that option really be named "--fs-type-exclude" instead? I may be mis-understandign things here...

[zevv]

Ico> --fs-exclude=fuse.sshfs Hmmm... should that option really be named "--fs-type-exclude" instead? I may be mis-understandign things here...

In hindsight that might have been a better name, but since changing options tends to confuse users I'm afraid we're stuck with this for now.

…

-- :wq ^X^Cy^K^X^C^C^C^C

[knedlsepp]

Thanks! Adding

[index]
fs-exclude fuse.sshfs

to the .ducrc file solved my particular problem. Having the possibility to do absolute excludes is however still a hot topic. ;-)

[l8gravely]

>>>> "Ico" == Ico Doornekamp ***@***.***> writes:

Ico> --fs-exclude=fuse.sshfs

> > Hmmm... should that option really be named "--fs-type-exclude" > instead? I may be mis-understandign things here...

Ico> In hindsight that might have been a better name, but since changing Ico> options tends to confuse users I'm afraid we're stuck with this for now. I don't think so. Just change it so that the old option is deprecated, and use two new ones.

…

--exclude and --fs-type-exclude Though maybe it's not a big deal now that I re-read the man page. It's pretty explicity what the difference is here. John

[graham-collinson]

Any reason not to just use realpath in match_exclude to get the full path? Easier than keeping track of the absolute path?
Or is there an overhead to using realpath that would make it better to keep track?

Something like:

static int match_exclude(struct duc *duc, const char *name, struct exclude *list)
{
	char path_full[DUC_PATH_MAX];
	char msg[DUC_PATH_MAX + 128];
	char *res = realpath(name, path_full);
	if (res == NULL) {
	    duc_log(duc, DUC_LOG_WRN, "Cannot determine realpath of: %s", name);
	}
	struct exclude *e;
	LL_FOREACH(list, e) {
#ifdef HAVE_FNMATCH_H
		if(fnmatch(e->name, path_full, 0) == 0) return 1;
#else
		if(strstr(path_full, e->name) == 0) return 1;
#endif
	}
	return 0;
}

[eliliam]

Bumping this because I am also having the same issue. I have multiple mounts of various types under /mnt that I do not care to index. However I would like to index everything else off of /, including other filesystems that are mounted, such as /home and /var/log. Is there a way to do this?

[vivasteve]

under /mnt that I do not care to index

if I'm understanding you correctly, this works for me to excluding /mnt:

duc index --exclude=mnt --exclude={proc,dev,run,sys,root} /

[vivasteve]

For some reason, for the life of me, I cannot get home to be excluded

duc index --exclude=home /

always includes home ...
(home is on a separate partition)