This project is mirrored from https://gitee.com/cowcomic/pixie.git.
Pull mirroring failed .
Repository mirroring has been paused due to too many failed attempts. It can be resumed by a project maintainer.
Repository mirroring has been paused due to too many failed attempts. It can be resumed by a project maintainer.
- 24 Feb, 2022 13 commits
-
-
James Bartlett authored
Summary: Adds a UDTF that calls ReleaseFreeMemory on all the agents. This causes tcmalloc to release spans in the free list to the OS. This is useful for experimenting with memory usage. Test Plan: Checked that px._HeapStats, showed about 600MB of free spans in the page heap, then after running the UDTF, another call to `px._HeapStats` showed that 500MB of that was returned to the OS. Reviewers: oazizi, vihang, nserrino Reviewed By: vihang Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10846 GitOrigin-RevId: 7acee822d2514fa562ea1895ab7bcee54e1b5f56
-
Vihang Mehta authored
Summary: `http.Error` sets headers but by the time `ResponseWriter.write` is called, headers are already sent. So we can't really do much with the error at that point. Just drop it instead. Test Plan: The sever shouldn't have logs saying ` http: superfluous response.WriteHeader call` anymore. Reviewers: jamesbartlett Reviewed By: jamesbartlett Signed-off-by:
Vihang Mehta <vihang@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10844 GitOrigin-RevId: 3886ac0ca8088ff7ee8bee85e97648a108df86e7
-
Phillip Kuznetsov authored
Summary: gendoc.py generated python docs with an extra nesting. That meant the python docs wouldn't get generated properly. We must've given up on this earlier which is why we didn't notice the problem. Test Plan: Generated the docs and made sure the python docs render from the data. Reviewers: htroisi, michelle, vihang Reviewed By: vihang JIRA Issues: PP-3156 Signed-off-by:
Phillip Kuznetsov <pkuznetsov@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10840 GitOrigin-RevId: f5018c03aa42aacd349ae4bc3934eef66c30440a
-
Pete Stevenson authored
Summary: The Java symbolization agent responds to a "method unload" callback. Here, we add support on the Stirling side for the same. Essentially, this removes a symbol from our Java symbol index. Test Plan: Existing. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: yzhao, oazizi JIRA Issues: PP-2982, PP-3264 Signed-off-by:
Pete Stevenson <jps@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10805 GitOrigin-RevId: 92db82567a1c91e608dbb507f10eafc3a4a53c7f
-
Pete Stevenson authored
Summary: Because of symbol unloading, keeping Java symbols cached is challenging. For now, make the Java symbolizer uncacheable. We do not anticipate significant loss of perf. because the Java symbolizer keeps its own symbol index in memory (i.e. is already like a symbol cache). Test Plan: Existing. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: yzhao, oazizi JIRA Issues: PP-2982, PP-3264 Signed-off-by:
Pete Stevenson <jps@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10821 GitOrigin-RevId: 9c6f1160c90357f3f9165003ec883f614c14e88e
-
Omid Azizi authored
Summary: Just a few typos and fixes noticed along the way. Test Plan: None. No functional changes. Reviewers: #stirling, jps Reviewed By: #stirling, jps Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10843 GitOrigin-RevId: 5e868e562cc785c754fa239d9595eddf7c6d5e60
-
James Bartlett authored
Summary: Adds alternative DataStreamBuffer implementation, which maintains events in a sorted map, until a call to `Head()` at which point the first contiguous events are merged into a contiguous buffer. This comes at the expense of an extra copy, but makes `RemovePrefix()` and future eviction work much faster, and also has much better memory properties when there are "gaps" in the data. Note that this diff removes `Get()` as a public API of DataStreamBuffer, and changes all call sites to use `Head()` instead (all call sites were in tests). Also changes Head to be non-const. Also adds a benchmark for the two DataStreamBufferImpl's. Using the socket tracer benchmark, and google/benchmark's compare.py tool, the following are the p values for the null hypothesis that the old and the new implementation are the same. These data suggest that the new implementation is statistically significantly faster on all benchmarks: ``` BM_SocketTraceConnector/http1_no_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/http1_encode_chunked_no_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/mysql_no_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/postgres_no_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/cql_no_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/nats_no_gaps_pvalue 0.0013 0.0013 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/http1_inter_iter_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/mysql_inter_iter_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/postgres_inter_iter_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/nats_inter_iter_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/cql_inter_iter_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 BM_SocketTraceConnector/http1_intra_iter_gaps_pvalue 0.0002 0.0002 U Test, Repetitions: 10 vs 10 ``` The mean on those benchmarks for the new implementation: ``` BM_SocketTraceConnector/http1_no_gaps_mean 8.90ms 22MB BM_SocketTraceConnector/http1_encode_chunked_no_gaps_mean 23.17ms 23MB BM_SocketTraceConnector/mysql_no_gaps_mean 12.02ms 21MB BM_SocketTraceConnector/postgres_no_gaps_mean 16.25ms 23MB BM_SocketTraceConnector/cql_no_gaps_mean 8.20ms 20MB BM_SocketTraceConnector/nats_no_gaps_mean 45.20ms 23MB BM_SocketTraceConnector/http1_inter_iter_gaps_mean 39.08ms 23MB BM_SocketTraceConnector/mysql_inter_iter_gaps_mean 72.62ms 23MB BM_SocketTraceConnector/postgres_inter_iter_gaps_mean 99.01ms 25MB BM_SocketTraceConnector/nats_inter_iter_gaps_mean 219.77ms 24MB BM_SocketTraceConnector/cql_inter_iter_gaps_mean 37.61ms 22MB BM_SocketTraceConnector/http1_intra_iter_gaps_mean 36.24ms 64MB ``` and for the old implementation: ``` BM_SocketTraceConnector/http1_no_gaps_mean 14.91ms 36MB BM_SocketTraceConnector/http1_encode_chunked_no_gaps_mean 39.66ms 39MB BM_SocketTraceConnector/mysql_no_gaps_mean 19.47ms 38MB BM_SocketTraceConnector/postgres_no_gaps_mean 31.09ms 39MB BM_SocketTraceConnector/cql_no_gaps_mean 19.58ms 36MB BM_SocketTraceConnector/nats_no_gaps_mean 51.89ms 40MB BM_SocketTraceConnector/http1_inter_iter_gaps_mean 90.18ms 63MB BM_SocketTraceConnector/mysql_inter_iter_gaps_mean 124.34ms 62MB BM_SocketTraceConnector/postgres_inter_iter_gaps_mean 160.59ms 64MB BM_SocketTraceConnector/nats_inter_iter_gaps_mean 299.13ms 44MB BM_SocketTraceConnector/cql_inter_iter_gaps_mean 84.91ms 62MB BM_SocketTraceConnector/http1_intra_iter_gaps_mean 312.37ms 527MB ``` Particularly of note is the dramatic improvement in memory usage for the `intra_iter_gaps` case from the new to the old. Test Plan: - Tested performance of new implementation in benchmarks, see above. - Tested correctness by making the data_stream_buffer_test test both the new and the old implementation. - Also tested that a skaffold deploy with the new DataStreamBuffer works on my dev cluster. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi, yzhao JIRA Issues: PP-3223 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10812 GitOrigin-RevId: a755ff6e7f288dece84193382cb9d9e4e1578ba0
-
James Bartlett authored
Summary: Adds a `RecordGenerator` for NATS data to the socket tracer benchmark. Blocked by D10800. Test Plan: No production changes. Tested that all the input NATS records get output by the benchmark. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Subscribers: yzhao JIRA Issues: PP-3211 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10801 GitOrigin-RevId: fd0e44b73e3271baa561412e86c3380617a1ae07
-
James Bartlett authored
Summary: Adds a `RecordGenerator` for cql data to the socket tracer benchmark. Blocked by D10799 Test Plan: No production changes. Tested that all the input cql records get output by the benchmark. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Subscribers: yzhao JIRA Issues: PP-3211 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10800 GitOrigin-RevId: 8e20b8dfe27de4c0e39d5b2adcd1d19d12e5b430
-
Vihang Mehta authored
Summary: D10804 broke the simple `wrk` based test. This fixes it. Test Plan: Used skaffold to deploy the server and client. Reviewers: jamesbartlett, nserrino, zasgar Reviewed By: jamesbartlett Signed-off-by:
Vihang Mehta <vihang@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10841 GitOrigin-RevId: a2ef5659923e0f3d5e3f0652e9322f3c35732c49
-
Nick Lanam authored
Summary: Skips testing Ctrl/Cmd+Enter until script execution diff lands. Test Plan: Run Cypress as normal (see `src/ui/cypress/README.md`). Reviewers: michelle, vihang Reviewed By: michelle JIRA Issues: PC-1443 Signed-off-by:
Nick Lanam <nlanam@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10816 GitOrigin-RevId: 249f4fa64e806bc5cf852112583d1eece67d2502
-
Nick Lanam authored
Summary: Adding a simple button that does this: easy. Making it look okay, not overlap the content, be obviously for the details, etc: a bit harder. This makes the details pane a bit more complex so that it can do this: - Details pane has a sticky header - Subtle shadow on header only when not scrolled to the top - Close button is clearly marked to be for the details - Table state syncs with close button correctly - Light and dark mode both look right {F179767} Test Plan: Run any PxL script that yields tables. Expand a row. Scroll around in the details. Expand other rows, press the close button, expand different rows. Expansion state should always match between the row highlight and the expansion pane. Try with both overlay and inline scrollbars (Chrome default vs Firefox default respectively; both can be configured to do the opposite by the end user). Reviewers: michelle, vihang Reviewed By: michelle JIRA Issues: PC-1444 Signed-off-by:
Nick Lanam <nlanam@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10838 GitOrigin-RevId: fc2931e8f59887811e18896d8cff87b29d9242af
-
Nick Lanam authored
Summary: TSIA. Test Plan: Run local UI. Visit `/live`, `/live/clusters`, and `/live/clusters/`. All should redirect to `/live/clusters/DEFAULT_CLUSTER_ID` if authenticated. Reviewers: michelle, vihang Reviewed By: michelle JIRA Issues: PC-1425 Signed-off-by:
Nick Lanam <nlanam@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10839 GitOrigin-RevId: 70a64c1876b1d894223cdd1d8f2ca6bc7da7026a
-
- 23 Feb, 2022 3 commits
-
-
Omid Azizi authored
Summary: Fix a seg-fault introduced in D10823. Test Plan: Manually deployed on to a cluster and left it running for a while. Previously, the seg-fault would trigger within minutes. This no longer happens. Reviewers: #stirling, michelle, yzhao Reviewed By: #stirling, yzhao Subscribers: yzhao Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10836 GitOrigin-RevId: ef3b3a4d444afa7778963fef889fbe2b09604426
-
Phillip Kuznetsov authored
Summary: We used to use the opname for the docs. But when I was reading the docs, I found this to be immensely confusing. I switched over to using the PxL name instead. This change was used to make the output docs: https://github.com/pixie-io/docs.px.dev/pull/143 Test Plan: Rendered the output docs in local dev. Reviewers: vihang, michelle, nserrino Reviewed By: michelle Signed-off-by:
Phillip Kuznetsov <pkuznetsov@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10819 GitOrigin-RevId: 48674fa9b715b579459769fe09b033a89632e7bf
-
Michelle Nguyen authored
Summary: Our licenses generator wasn't able to pick these up, so adding them manually. Test Plan: Ran license generator Reviewers: vihang Reviewed By: vihang Signed-off-by:
Michelle Nguyen <michellenguyen@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10833 GitOrigin-RevId: cde0ba0fcc3ca0a3d149de2c97211f29d4fbead9
-
- 22 Feb, 2022 5 commits
-
-
Omid Azizi authored
Summary: absl::bind_front has some advantages over std::bind. See https://abseil.io/tips/108 . C++20 introduces std::bind_front() too, so we can switch to that once we are using it. Test Plan: No functional changes - Existing tests. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10824 GitOrigin-RevId: ce2909deb26cc71bb557fdeafc0635edc9bba0e2
-
Omid Azizi authored
Summary: A config flag controls whether we scan for mmap events, which is in turn used to detect dynamically loaded libraries. When this flag is off, there could be a leak, since we add to the map but never drain it. In this diff, we make sure not to add any items to the map if the flag has disabled the feature. Test Plan: Rely on existing tests to ensure no functionality is broken. Manual check for the leak fix. Reviewers: #stirling, jamesbartlett Reviewed By: jamesbartlett Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10830 GitOrigin-RevId: b5bd65214fbe9d7f17f7ea1edb4cd779828af022
-
Omid Azizi authored
Summary: The custom decoder is faster by ~7% ``` ---------------------------------------------------------------- Benchmark Time CPU Iterations ---------------------------------------------------------------- BM_custom_body_parser 58485 ns 58480 ns 10275 BM_pico_body_parser 62858 ns 62841 ns 11012 ``` Test Plan: There's already test coverage for this. Just flipping the switch. Reviewers: #stirling, jamesbartlett, yzhao Reviewed By: #stirling, yzhao Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10825 GitOrigin-RevId: 97694b23e8f9a3f7dc9a8bba7e6e10122d95d584
-
Omid Azizi authored
Summary: In the rare case when a container process switches namespaces with a call to setns, NetworkStatsConnector can incorrectly produce stats from the wrong namespace. Mitigate this effect by choosing the oldest PID in the container. This provides stability in which PID is chosen for producing stats. We also assume here that the oldest PID is not the one switching namespaces, so a more robust fix may be required in the future. Note that this bug was found by monitoring the PEM itself. Stirling forks a Java agent attach process which then changes namespaces, causing mis-reported network stats. Test Plan: Test added for the ordering. Also checked manually that the mis-reported network stats are fixed. Reviewers: #stirling, michelle, jps Reviewed By: #stirling, jps Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10823 GitOrigin-RevId: 021a2d1d77805a7713a4e9a2acfae35c57a14255
-
Omid Azizi authored
Summary: This is a test utility for limiting Stirling's monitoring to a smaller set of PIDs. Test Plan: This is test tooling, so rely on existing tests. Reviewers: #stirling, jps Reviewed By: #stirling, jps Subscribers: jps Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10827 GitOrigin-RevId: 47641e1848126bedfd6b8ce9b20862f22fe9f556
-
- 21 Feb, 2022 1 commit
-
-
Omid Azizi authored
Summary: Add a test to ensure only the desired namepsace is affected. Test Plan: This is a test. Reviewers: #stirling, jps Reviewed By: #stirling, jps Subscribers: jps Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10815 GitOrigin-RevId: f5a0738bdb65b27c0b66ad2cf848b475cf9b6e09
-
- 18 Feb, 2022 7 commits
-
-
James Bartlett authored
Summary: Adds a `RecordGenerator` for mysql data. Blocked by D10798 Test Plan: No production changes. Tested that all the input mysql records get output by the benchmark. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi JIRA Issues: PP-3211 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10799 GitOrigin-RevId: c7042be8d4ea007ee3318a77f77976ca344087bd
-
Pete Stevenson authored
PerfProfiler: change args to java agent to indicate the symbol file path (vs. name stem, which was awkward). Summary: Previously, the symbolization agent args indicated a file name stem because the symbol file name was uniquified based on target process pid. The name was given as a name stem (in case the agent needed to create a different file name for the txt log file). This setup was awkward both because it was a name stem and because it required coordination of the symbol file name in several places. Now we uniquify based on injecting the target process pid into the artifacts path. In this diff, we change the JVMTI symbolization agent args to indicate only the artifacts path and place a standard symbol file name in the shared header file. Test Plan: Existing. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi JIRA Issues: PP-2982 Signed-off-by:
Pete Stevenson <jps@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10818 GitOrigin-RevId: 902ec31bb1f306ded30f5e9714801c306207506c
-
James Bartlett authored
DataStreamBuffer: move the implementation of DataStreamBuffer behind a private implementation class. Summary: This diff allows for a runtime flag to switch which DataStreamBuffer implementation to use. Currently, there's only the one implementation, but a future diff will add a new DataStreamBuffer implementation. Test Plan: This diff is mostly moving code around, but to be safe I tested: - Existing data stream buffer tests pass - Skaffold deploy works, and data can be seen as usual. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi, yzhao Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10802 GitOrigin-RevId: 54f35a4df4b04d3dbe6225458fb1d1e9beff39c2
-
James Bartlett authored
Summary: Adds a `RecordGenerator` that generates postgres data. Blocked by D10797 Test Plan: No production changes. Tested that the benchmark outputs all the input records for the new postgres data. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi JIRA Issues: PP-3211 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10798 GitOrigin-RevId: 5594ea97de0e8815af417d09fe1f8386b6981271
-
James Bartlett authored
Summary: Adds two kinds of `PosGenerator`'s that generate "gappy" data (i.e. simulate data from BPF that has large gaps in the data stream): - One generates data that has gaps between simulated polling iterations. - One generates data that has gaps within a single polling iteration. Blocked by D10793 Test Plan: No production changes. Tested that the new PosGenerator's work with the benchmark. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi JIRA Issues: PP-3211 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10797 GitOrigin-RevId: 5f85395d1e06c7843e50836903a7075b56645f93
-
Pete Stevenson authored
Summary: We included <utility> based on a complaint from our linter, but the code subsequently changed leaving a stale include. Here, we remove that. Test Plan: Build process. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi JIRA Issues: PP-2982 Signed-off-by:
Pete Stevenson <jps@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10811 GitOrigin-RevId: 8d74f0747abe2daea37e634db28863ae25837d21
-
James Bartlett authored
Summary: When we moved the retention size limit restriction from DataStream::ProcessBytesToFrames into CleanupEvents, we neglected to also move the ShrinkToFit so that the DSB would be shrunk to the retention size. This diff just moves the ShrinkToFit into CleanupEvents. No need to still call ShrinkToFit in ProcessBytesToFrames because CleanupEvents will be called briefly after the end of ProcessBytesToFrames no matter what. Test Plan: Existing tests pass. Tried it out in skaffold and made sure scripts work as normal. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10813 GitOrigin-RevId: ef4a1e4aa97f56573d9b53b3bcc6530617a0f8dd
-
- 17 Feb, 2022 11 commits
-
-
Phillip Kuznetsov authored
Summary: Simple fix. Something I spotted while documenting the data pipelines. Test Plan: Unit tests pass Reviewers: michelle, nserrino Reviewed By: michelle Signed-off-by:
Phillip Kuznetsov <pkuznetsov@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10814 GitOrigin-RevId: b33a01cf0024d2dc21b02e71d9f10a02eb7d16ab
-
Omid Azizi authored
Summary: Update the Mux protocol ByteSize() function to be more accurate. Test Plan: TBD. Reviewers: #stirling, rcheng Reviewed By: #stirling, rcheng Subscribers: rcheng Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10745 GitOrigin-RevId: 0ae374ae50b05e71b5c3a6b3dbca5b042e4a33b9
-
Omid Azizi authored
Summary: Use more inclusive language. Test Plan: This is just a name change. Rely on existing tests. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10808 GitOrigin-RevId: 9f3c06a144a838e09347adf6528eba8da49e9387
-
James Bartlett authored
Summary: - Added a data generation framework, that separates the generation of the data from the generation of the `pos` byte label for each event. Users of the data generation framework specify a `RecordGenerator` and a `PosGenerator`. The `RecordGenerator` has a single method `Next` that should return a record (a collection of traffic direction, frame pairs) that represents a single record that would be pushed to the TableStore if successfully parsed. The `PosGenerator` has a method that returns the byte position for the next event given the next event's size, and a method to signal the next polling iteration of data (useful for adding a gap in byte position between simulated polling iteration). Using the provided `PosGenerator` and `RecordGenerator` the data generation framework creates events per connection per simulated polling iteration (RecordGenerators are instantiated per connection and PosGenerators are instantiated per direction per connection). - To keep this diff as small as possible I only added a minimal set of Pos and Record generators in order to benchmark only HTTP1 traffic with continuous `pos` labels. Future diffs add more interesting PosGenerators as well as simple RecordGenerators for a variety of protocols. - Added the benchmark itself. The benchmark first generates all the necessary data using the above framework (as specified by `BenchmarkDataGenerationSpec`). Then during a single benchmark iteration, it will for each simulated "polling" iteration: push each event for that iteration to the SocketTraceConnector as if it had come from BPF (eg. via HandleDataEvent), then call `TransferData` once. During the first benchmark iteration the `MemoryTracker` will be enabled to keep track of starting, ending and peak memory usage for that particular benchmark. Finally after all benchmark iterations, the benchmark reports a wide variety of statistics. The statistics reported are: `PollIters` (the number of simulated polling iterations), `AllocPeak` (the maximum memory used during the benchmark minus the memory used before the benchmark started processing data), `PhysMem{Start,End}` (The physical memory used at the start/end of the benchmark, physical memory includes memory that the program has freed to the allocator but the allocator hasn't released to the OS), `AllocMem{Start,End}` (The memory that has not been freed to the allocator at the start/end of the benchmark), `EquivalendWorkloadThroughput` (The throughput in B/s of a workload that would produce similar load to this benchmark), `Bytes{Input/Output}`/`Records{Input/Output}` (The amount of bytes/records of data input/output to/from the benchmark, note that records input should equal records output if no data was lost), and finally `NumEvents` (the total number of events processed). - Because there are so many statistics, I added a variety of flags to control what statistics get printed, by default only `AllocPeak` and `PollIters` are displayed. Blocked by D10786, D10784, D10750 Test Plan: No production changes, tested that the new benchmarking code runs the benchmark successfully, and that for each protocol the generated data can be successfully parsed. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: yzhao, oazizi JIRA Issues: PP-3211 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10793 GitOrigin-RevId: b0e62c9dbdb3b9e6f70691d127bf74b1b207a78e
-
Natalie Serrino authored
Summary: I tried to do a px.abs on the result of px.parse_duration, but noticed that it didn't preserve the DURATION_NS type. This diff adds the correct semantic type to the result of px.abs. Test Plan: not needed, we use these same rules elsewhere, just need to add them here Reviewers: philkuz, jamesbartlett Reviewed By: jamesbartlett Signed-off-by:
Natalie Serrino <nserrino@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10809 GitOrigin-RevId: d7a22eac106a04fa7ac9c806457b594f18b1887d
-
yzhao1012 authored
Summary: This allows skaffold to change PEM source connectors without making code changes Test Plan: Tested with skaffold Reviewers: #stirling, jps Reviewed By: #stirling, jps Subscribers: oazizi Signed-off-by:
yzhao1012 <yzhao@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10794 GitOrigin-RevId: 637ba96e276b1a5c3edecd1d16a4e36569fb9400
-
yzhao1012 authored
Summary: One example of improving organization. Similar headers, like system_headers.{cc,h} are already in src/stirling/utils. Test Plan: Move code files, Jenkins build covers this Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Signed-off-by:
yzhao1012 <yzhao@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10806 GitOrigin-RevId: 3f869ba454c22b363f428ea7614bfd17cd6246f2
-
James Bartlett authored
Summary: For the purposes of the benchmark, we want to be able to generate a bunch of wire data for each protocol. This diff adds the utilities to make data generation easier. These utilities could also be used in the tests instead of the ad-hoc char[] byte buffers that the tests currently used. This diff adds a couple examples in the tests to demonstrate this usage. Note, this diff also restructures the build file so that test utilities don't get built into the main binary. Test Plan: No functional changes, so relying on existing tests. This diff also increases test coverage. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi, yzhao JIRA Issues: PP-3211 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10784 GitOrigin-RevId: 9317f1562fc37ce1a7155599e7f8d493456a20d2
-
Omid Azizi authored
Summary: The old implementation of DNS' ByteSize() used a const_cast which can be avoided. This diff cleans that up. Test Plan: No functional change, so existing tests. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Subscribers: yzhao Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10776 GitOrigin-RevId: 802c504f386745585438be885b9888ab32d35edc
-
James Bartlett authored
Summary: For the purposes of the socket tracer benchmark, we want to be able to generate a bunch of wire data for each protocol. This diff adds the utilities to make data generation easier for pgsql. These utilities could also be used in the tests instead of the ad-hoc char[] byte buffers that the tests currently used. I added a couple examples in the tests to demonstrate this usage. This diff also restructures the build file, such that the test data is not compiled with the main cc library. Test Plan: No functional changes, relying on existing tests. Note that this diff increases test coverage. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Subscribers: yzhao JIRA Issues: PP-3211 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10786 GitOrigin-RevId: 6d379b8aac9964bc27959f0378483d29d911409b
-
Vihang Mehta authored
Summary: Use POST message type and JSON parse params. Test Plan: `curl -X POST -vvv --raw -H 'Content-Type: application/json' --data '{"chunk_size":10, "body_size":100}' http://localhost:8111/` Reviewers: zasgar, nserrino Reviewed By: zasgar Signed-off-by:
Vihang Mehta <vihang@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10804 GitOrigin-RevId: fea928b276a065840723681a097042a5696a11a5
-