New versions of Collectl?

Laurence Oberman — Tue, 02 Jan 2024 14:54:02 -0000

Happy New Year
Please not, there will no longer be version updates on Sourceforge.

All updates will be github only
git clone https://github.com/sharkcz/collectl.git

For problem reports please email loberman@redhat.com. Include a full description of the issue.
Thanks
Laurence

New versions of Collectl?

Laurence Oberman — Tue, 30 May 2023 18:41:42 -0000

Hello

Indeed if you have been watching sourceforge, I tried to keep it updated in multiple places but now have started primarily updating on github.

git clone https://github.com/sharkcz/collectl.git
cd coll*
./INSTALL

For RHEL7 and higher:

systemctl enable collectl
systemctl start collectl

Note that I do not have time to test on all different distros so if anything comes up let me know on Sourceforge or email me here.
I still watch Sourceforge for problem reports.

Regards
Laurence Oberman

New versions of Collectl?

Ben Fulton — Fri, 26 May 2023 18:03:06 -0000

Hi,

It looks like this project is being supported again? Is there any kind of roadmap or documentation update for it? The website says the last update was in 2018.

collectl lustre 4.3.1 on lustre 2.10 is working nicely

windy — Tue, 04 Apr 2023 01:58:00 -0000

thank you. I use it that works well on 4.3.8 with lustre 2.10.2 . will this pacth add to master version?

Collectl 4.3.8 released today

Laurence Oberman — Tue, 07 Feb 2023 18:58:06 -0000

Note that it is as always, also available to be cloned here
https://github.com/sharkcz/collectl.git

Collectl 4.3.8 released today

Laurence Oberman — Tue, 07 Feb 2023 18:56:56 -0000

Added rbd devices
Added additional network devices names
Updated version number (forgot to do that on the last release)

collectl-4.3.7.src.tar.gz was released today

Laurence Oberman — Thu, 26 Jan 2023 15:40:02 -0000

collectl-4.3.7.src.tar.gz
Released with an fix to include nvme devices that have native multipathing.

commit cd8a94b61f5700945af8f4e6a3fcd0e4ea18f1e2 (HEAD -> master, origin/master, origin/HEAD)
Author: Laurence Oberman loberman@redhat.com
Date: Wed Jan 25 17:21:32 2023 -0500

Add the nvme native multipath devices filter to the collectl.conf
"nvme\d+c\d+"

Also available here
got clone https://github.com/sharkcz/collectl.git

vmstatInit fails on second file when using wildcards

Bill Torpey — Thu, 01 Dec 2022 17:15:37 -0000

Sorry if that wasn't clear: we run collectl once to get per-process cpu utilization (--procopts kw4096 -sZ -oDG), a second time to get per-process memory utilization (--procopts mkw4096 -sZ -oDG), a third time to get system-wide network utilization (-sN -oD), and fourth to get vmstat (--export vmstat -oD). All of that is from the same .raw file(s).

Again, we did it that way just so we could run both collectl and our old monitoring tools and compare like-for-like.

We'll be changing that to generate plot-format files from the raw files, which means we'll only need to pass the raw file once. That's a win: for this 5-day test it takes >20 minutes for a single pass of the four raw files which total a bit over 10GB combined. (This on a CentOS 7 box with 4/8(HT) x i7-4770R CPU @ 3.20GHz, 16GB RAM and SSD. It's a bit quicker when we're not asking for per-process stats). Multiply that by 10 hosts and a full run takes a while.

We specify a procfilt in our etc/collectl/conf so only capture processes that we care about -- unfortunately for some of them we need to filter on the full command line. We are sampling once per-second, which takes very little overhead on the collection side, but it does impact post-processing.

And yes, the command lines we capture/process are ridiculously large (4k) -- blame java and/or people who put all their jar's on the command line instead of in an environment variable. But we need the full command line to filter the proper processes.

Last but not least we can't use colplot directly -- we already have a set of standard gnuplot templates that we use, some of which pull data from multiple files (e.g., to compare latency against message rate). Not all data comes from collectl either -- we pull data from application logs and combine it with things like cpu and memory.

Again, thanks for collectl and for all your help!

vmstatInit fails on second file when using wildcards

Bill Torpey — Thu, 01 Dec 2022 15:55:46 -0000

Thanks Laurence, I'll check that out. FWIW, attached is the kind of output we get from vmstat.

vmstatInit fails on second file when using wildcards

Bill Torpey — Thu, 01 Dec 2022 15:35:44 -0000

Hi Mark (and Laurence):

First off, thanks for all the help, and for a great tool! Please see replies inline below:

Bill - I guess the one overall question I'd ask is about your general use of collectl and why you have a handful of smallisk files you need to play back together and also way none of your files are compressed?

We've recently switched to collectl for monitoring our QA performance testing environment, with a view to rolling out collectl in production. collectl is replacing a bunch of custom scripts that basically just ran top, vmstat, etc. in a loop and dumped output to files that were processed by a set of Perl scripts. Those scripts consolidated data across a dozen or so hosts to give a high-level view of system utilization at different message rates, and helps us identify bottlenecks and performance regressions in our application.

The first pass is a one-for-one replacement of the existing scripts, and for that reason it ends up running collectl four times: once each for cpu (per-process), memory (also per-process), network and vmstat-type output (block writes, iowait, etc.).

It's MUCH faster to unzip the file(s) first, rather than have collectl read the gzipped file directly, even once and obviously four times.

The next step would/should be to rewrite the original Perl scripts to take collectl's plot format, in which case we'll only need to read the raw file once. (I've also just finished splitting up the scripts so we can run the collectl analysis in background for each host separately).

I had designed collectl to very efficient in terms of cpu and storage. When I installed collectl on any system, I'd just turn it on and leave it running forever. For that reason, files are automatically purged after a week (configurable of course).

That's exactly what we plan to do in production. For our QA performance tests, it makes more sense to start and stop collectl along with our test harness. We archive the (compressed) collectl files along with application logs, etc. that we use for running our analysis on, and doing it this way means the collectl files are smaller (most performance tests only run for an hour or two).

When I want to diagnose a problem, I'd first plot everything with colplot, selecting timeframes of interest. Then if I wanted more details I'd play back the file for that day with the appropriate form/thru switches. Colplot is certainly optional, but I'd find it to be most useful when I wasn't sure what I was looking for.

We have custom gnuplot scripts that generate specific plots that we use a lot, but I would like to use colplot once we deploy to production for real-time trouble-shooting, just haven't gotten there yet.

As for compressed files, when I first wrote colplot, you had to have text based files to feed into gnuplot. later on, gnuplot supported plotting compressed files and to my delight was almost as efficient. I'm guessing because the files were something like 10% the size and so much less I/O required to read them.

Interesting — I didn't know that. That might be helpful, since some of our files do get a bit large. For instance, the specific test that I'm currently analyzing is a "5-day test", meant to model a full week of trading activity, and took three days real-time to run. The latency data file for this test has about 75 million rows, which ends up being around 8GB.

In any event, I'm always happy to here collectl lives on and to see people using it in different ways. As it turns out I think you're the first person I remember who found vmstat format useful, possibly why that bug you uncovered took over 20 years to find! ;)

Happy to help ;-) Seriously, I'm a little surprised — for our db servers we care a lot about block writes and iowait, and afaict the vmstat export is the only way to get that from collectl.

In any event, thanks again for a really cool tool. It's taken me several years to get it deployed in our environment, but it's absolutely terrific.

Last but not least — any hints or tips for speeding up post-processing analysis would be appreciated — collectl is very efficient in collecting, but generating reports from the collected data can take quite a while.

Recent posts to Discussion

New versions of Collectl?

New versions of Collectl?

New versions of Collectl?

collectl lustre 4.3.1 on lustre 2.10 is working nicely

Collectl 4.3.8 released today

Collectl 4.3.8 released today

collectl-4.3.7.src.tar.gz was released today

vmstatInit fails on second file when using wildcards

vmstatInit fails on second file when using wildcards

vmstatInit fails on second file when using wildcards