This project is mirrored from https://gitee.com/cowcomic/pixie.git. Pull mirroring failed .
Repository mirroring has been paused due to too many failed attempts. It can be resumed by a project maintainer.
  1. 24 Feb, 2022 13 commits
    • James Bartlett's avatar
      Add px._HeapReleaseFreeMemory UDTF. · 56bdd209
      James Bartlett authored
      
      Summary:
      Adds a UDTF that calls ReleaseFreeMemory on all the agents.
      This causes tcmalloc to release spans in the free list to the OS. This is useful for experimenting with memory usage.
      
      Test Plan: Checked that px._HeapStats, showed about 600MB of free spans in the page heap, then after running the UDTF, another call to `px._HeapStats` showed that 500MB of that was returned to the OS.
      
      Reviewers: oazizi, vihang, nserrino
      
      Reviewed By: vihang
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10846
      
      GitOrigin-RevId: 7acee822d2514fa562ea1895ab7bcee54e1b5f56
      56bdd209
    • Vihang Mehta's avatar
      Fix loadtest superfluous WriteHeader · 3df4ce33
      Vihang Mehta authored
      
      Summary:
      `http.Error` sets headers but by the time `ResponseWriter.write` is
      called, headers are already sent. So we can't really do much with the error at
      that point. Just drop it instead.
      
      Test Plan:
      The sever shouldn't have logs saying
      ` http: superfluous response.WriteHeader call` anymore.
      
      Reviewers: jamesbartlett
      
      Reviewed By: jamesbartlett
      Signed-off-by: default avatarVihang Mehta <vihang@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10844
      
      GitOrigin-RevId: 3886ac0ca8088ff7ee8bee85e97648a108df86e7
      3df4ce33
    • Phillip Kuznetsov's avatar
      [PP-3156] Fix output format for methods in python docs · 3ae2132b
      Phillip Kuznetsov authored
      
      Summary:
      gendoc.py generated python docs with an extra nesting. That meant the python docs wouldn't
      get generated properly. We must've given up on this earlier which is why we didn't notice the
      problem.
      
      Test Plan: Generated the docs and made sure the python docs render from the data.
      
      Reviewers: htroisi, michelle, vihang
      
      Reviewed By: vihang
      
      JIRA Issues: PP-3156
      Signed-off-by: default avatarPhillip Kuznetsov <pkuznetsov@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10840
      
      GitOrigin-RevId: f5018c03aa42aacd349ae4bc3934eef66c30440a
      3ae2132b
    • Pete Stevenson's avatar
      PerfProfiler: handle symbol unload on Stirling side of Java symbolizer. · 431a4762
      Pete Stevenson authored
      
      Summary: The Java symbolization agent responds to a "method unload" callback. Here, we add support on the Stirling side for the same. Essentially, this removes a symbol from our Java symbol index.
      
      Test Plan: Existing.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      Subscribers: yzhao, oazizi
      
      JIRA Issues: PP-2982, PP-3264
      Signed-off-by: default avatarPete Stevenson <jps@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10805
      
      GitOrigin-RevId: 92db82567a1c91e608dbb507f10eafc3a4a53c7f
      431a4762
    • Pete Stevenson's avatar
      PerfProfiler: make the Java symbolizer uncacheable. · 102988b6
      Pete Stevenson authored
      
      Summary: Because of symbol unloading, keeping Java symbols cached is challenging. For now, make the Java symbolizer uncacheable. We do not anticipate significant loss of perf. because the Java symbolizer keeps its own symbol index in memory (i.e. is already like a symbol cache).
      
      Test Plan: Existing.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      Subscribers: yzhao, oazizi
      
      JIRA Issues: PP-2982, PP-3264
      Signed-off-by: default avatarPete Stevenson <jps@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10821
      
      GitOrigin-RevId: 9c6f1160c90357f3f9165003ec883f614c14e88e
      102988b6
    • Omid Azizi's avatar
      Fix some comments · 1c25fbde
      Omid Azizi authored
      
      Summary: Just a few typos and fixes noticed along the way.
      
      Test Plan: None. No functional changes.
      
      Reviewers: #stirling, jps
      
      Reviewed By: #stirling, jps
      Signed-off-by: default avatarOmid Azizi <oazizi@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10843
      
      GitOrigin-RevId: 5e868e562cc785c754fa239d9595eddf7c6d5e60
      1c25fbde
    • James Bartlett's avatar
      DataStreamBuffer: Add alternative DataStreamBuffer implementation. · 41582918
      James Bartlett authored
      
      Summary:
      Adds alternative DataStreamBuffer implementation, which maintains events in a sorted map, until a call to `Head()` at which point the first contiguous events are merged into a contiguous buffer. This comes at the expense of an extra copy, but makes `RemovePrefix()` and future eviction work much faster, and also has much better memory properties when there are "gaps" in the data.
      
      Note that this diff removes `Get()` as a public API of DataStreamBuffer, and changes all call sites to use `Head()` instead (all call sites were in tests). Also changes Head to be non-const.
      
      Also adds a benchmark for the two DataStreamBufferImpl's.
      
      Using the socket tracer benchmark, and google/benchmark's compare.py tool, the following are the p values for the null hypothesis that the old and the new implementation are the same. These data suggest that the new implementation is statistically significantly faster on all benchmarks:
      ```
      BM_SocketTraceConnector/http1_no_gaps_pvalue                                0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/http1_encode_chunked_no_gaps_pvalue                 0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/mysql_no_gaps_pvalue                                0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/postgres_no_gaps_pvalue                             0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/cql_no_gaps_pvalue                                  0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/nats_no_gaps_pvalue                                 0.0013          0.0013      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/http1_inter_iter_gaps_pvalue                        0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/mysql_inter_iter_gaps_pvalue                        0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/postgres_inter_iter_gaps_pvalue                     0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/nats_inter_iter_gaps_pvalue                         0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/cql_inter_iter_gaps_pvalue                          0.0002          0.0002      U Test, Repetitions: 10 vs 10
      BM_SocketTraceConnector/http1_intra_iter_gaps_pvalue                        0.0002          0.0002      U Test, Repetitions: 10 vs 10
      ```
      The mean on those benchmarks for the new implementation:
      ```
      BM_SocketTraceConnector/http1_no_gaps_mean                 8.90ms    22MB
      BM_SocketTraceConnector/http1_encode_chunked_no_gaps_mean  23.17ms   23MB
      BM_SocketTraceConnector/mysql_no_gaps_mean                 12.02ms   21MB
      BM_SocketTraceConnector/postgres_no_gaps_mean              16.25ms   23MB
      BM_SocketTraceConnector/cql_no_gaps_mean                   8.20ms    20MB
      BM_SocketTraceConnector/nats_no_gaps_mean                  45.20ms   23MB
      BM_SocketTraceConnector/http1_inter_iter_gaps_mean         39.08ms   23MB
      BM_SocketTraceConnector/mysql_inter_iter_gaps_mean         72.62ms   23MB
      BM_SocketTraceConnector/postgres_inter_iter_gaps_mean      99.01ms   25MB
      BM_SocketTraceConnector/nats_inter_iter_gaps_mean          219.77ms  24MB
      BM_SocketTraceConnector/cql_inter_iter_gaps_mean           37.61ms   22MB
      BM_SocketTraceConnector/http1_intra_iter_gaps_mean         36.24ms   64MB
      ```
      and for the old implementation:
      ```
      BM_SocketTraceConnector/http1_no_gaps_mean                 14.91ms   36MB
      BM_SocketTraceConnector/http1_encode_chunked_no_gaps_mean  39.66ms   39MB
      BM_SocketTraceConnector/mysql_no_gaps_mean                 19.47ms   38MB
      BM_SocketTraceConnector/postgres_no_gaps_mean              31.09ms   39MB
      BM_SocketTraceConnector/cql_no_gaps_mean                   19.58ms   36MB
      BM_SocketTraceConnector/nats_no_gaps_mean                  51.89ms   40MB
      BM_SocketTraceConnector/http1_inter_iter_gaps_mean         90.18ms   63MB
      BM_SocketTraceConnector/mysql_inter_iter_gaps_mean         124.34ms  62MB
      BM_SocketTraceConnector/postgres_inter_iter_gaps_mean      160.59ms  64MB
      BM_SocketTraceConnector/nats_inter_iter_gaps_mean          299.13ms  44MB
      BM_SocketTraceConnector/cql_inter_iter_gaps_mean           84.91ms   62MB
      BM_SocketTraceConnector/http1_intra_iter_gaps_mean         312.37ms  527MB
      ```
      
      Particularly of note is the dramatic improvement in memory usage for the `intra_iter_gaps` case from the new to the old.
      
      Test Plan:
      - Tested performance of new implementation in benchmarks, see above.
      - Tested correctness by making the data_stream_buffer_test test both the new and the old implementation.
      - Also tested that a skaffold deploy with the new DataStreamBuffer works on my dev cluster.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      Subscribers: oazizi, yzhao
      
      JIRA Issues: PP-3223
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10812
      
      GitOrigin-RevId: a755ff6e7f288dece84193382cb9d9e4e1578ba0
      41582918
    • James Bartlett's avatar
      [PP-3211] Add NATS data to socket tracer benchmark. · 31dc4661
      James Bartlett authored
      
      Summary:
      Adds a `RecordGenerator` for NATS data to the socket tracer benchmark.
      
      Blocked by D10800.
      
      Test Plan:
      No production changes.
      Tested that all the input NATS records get output by the benchmark.
      
      Reviewers: #stirling, yzhao
      
      Reviewed By: #stirling, yzhao
      
      Subscribers: yzhao
      
      JIRA Issues: PP-3211
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10801
      
      GitOrigin-RevId: fd0e44b73e3271baa561412e86c3380617a1ae07
      31dc4661
    • James Bartlett's avatar
      [PP-3211] Add cql data to socket tracer benchmark. · 74166561
      James Bartlett authored
      
      Summary:
      Adds a `RecordGenerator` for cql data to the socket tracer benchmark.
      
      Blocked by D10799
      
      Test Plan:
      No production changes.
      Tested that all the input cql records get output by the benchmark.
      
      Reviewers: #stirling, yzhao
      
      Reviewed By: #stirling, yzhao
      
      Subscribers: yzhao
      
      JIRA Issues: PP-3211
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10800
      
      GitOrigin-RevId: 8e20b8dfe27de4c0e39d5b2adcd1d19d12e5b430
      74166561
    • Vihang Mehta's avatar
      Make wrk work with POST · be6a485e
      Vihang Mehta authored
      
      Summary: D10804 broke the simple `wrk` based test. This fixes it.
      
      Test Plan: Used skaffold to deploy the server and client.
      
      Reviewers: jamesbartlett, nserrino, zasgar
      
      Reviewed By: jamesbartlett
      Signed-off-by: default avatarVihang Mehta <vihang@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10841
      
      GitOrigin-RevId: a2ef5659923e0f3d5e3f0652e9322f3c35732c49
      be6a485e
    • Nick Lanam's avatar
      [PC-1443]: Cypress tests: keyboard shortcuts · f7769d92
      Nick Lanam authored
      
      Summary: Skips testing Ctrl/Cmd+Enter until script execution diff lands.
      
      Test Plan: Run Cypress as normal (see `src/ui/cypress/README.md`).
      
      Reviewers: michelle, vihang
      
      Reviewed By: michelle
      
      JIRA Issues: PC-1443
      Signed-off-by: default avatarNick Lanam <nlanam@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10816
      
      GitOrigin-RevId: 249f4fa64e806bc5cf852112583d1eece67d2502
      f7769d92
    • Nick Lanam's avatar
      [PC-1444]: Add close button to expanded table rows · 970e62b9
      Nick Lanam authored
      
      Summary:
      Adding a simple button that does this: easy.
      Making it look okay, not overlap the content, be obviously for the details, etc: a bit harder.
      
      This makes the details pane a bit more complex so that it can do this:
      - Details pane has a sticky header
      - Subtle shadow on header only when not scrolled to the top
      - Close button is clearly marked to be for the details
      - Table state syncs with close button correctly
      - Light and dark mode both look right
      
      {F179767}
      
      Test Plan:
      Run any PxL script that yields tables.
      Expand a row.
      Scroll around in the details.
      Expand other rows, press the close button, expand different rows.
      Expansion state should always match between the row highlight and the expansion pane.
      Try with both overlay and inline scrollbars (Chrome default vs Firefox default respectively; both can be configured to do the opposite by the end user).
      
      Reviewers: michelle, vihang
      
      Reviewed By: michelle
      
      JIRA Issues: PC-1444
      Signed-off-by: default avatarNick Lanam <nlanam@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10838
      
      GitOrigin-RevId: fc2931e8f59887811e18896d8cff87b29d9242af
      970e62b9
    • Nick Lanam's avatar
      [PC-1425]: Redirect `/clusters` to default cluster · 534d5ee6
      Nick Lanam authored
      
      Summary: TSIA.
      
      Test Plan:
      Run local UI.
      Visit `/live`, `/live/clusters`, and `/live/clusters/`. All should redirect to `/live/clusters/DEFAULT_CLUSTER_ID` if authenticated.
      
      Reviewers: michelle, vihang
      
      Reviewed By: michelle
      
      JIRA Issues: PC-1425
      Signed-off-by: default avatarNick Lanam <nlanam@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10839
      
      GitOrigin-RevId: 70a64c1876b1d894223cdd1d8f2ca6bc7da7026a
      534d5ee6
  2. 23 Feb, 2022 3 commits
  3. 22 Feb, 2022 5 commits
    • Omid Azizi's avatar
      Stirling: Use absl::bind_front instead of std::bind · 7a59e368
      Omid Azizi authored
      Summary:
      absl::bind_front has some advantages over std::bind. See https://abseil.io/tips/108
      
      .
      
      C++20 introduces std::bind_front() too, so we can switch to that once we are using it.
      
      Test Plan: No functional changes - Existing tests.
      
      Reviewers: #stirling, yzhao
      
      Reviewed By: #stirling, yzhao
      Signed-off-by: default avatarOmid Azizi <oazizi@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10824
      
      GitOrigin-RevId: ce2909deb26cc71bb557fdeafc0635edc9bba0e2
      7a59e368
    • Omid Azizi's avatar
      Memory leak fix in mmap event tracker structure · c4f19a20
      Omid Azizi authored
      
      Summary:
      A config flag controls whether we scan for mmap events, which is in turn used to detect dynamically loaded libraries.
      
      When this flag is off, there could be a leak, since we add to the map but never drain it.
      
      In this diff, we make sure not to add any items to the map if the flag has disabled the feature.
      
      Test Plan: Rely on existing tests to ensure no functionality is broken. Manual check for the leak fix.
      
      Reviewers: #stirling, jamesbartlett
      
      Reviewed By: jamesbartlett
      Signed-off-by: default avatarOmid Azizi <oazizi@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10830
      
      GitOrigin-RevId: b5bd65214fbe9d7f17f7ea1edb4cd779828af022
      c4f19a20
    • Omid Azizi's avatar
      Switch to custom chunked HTTP decoder, which is faster · 4d96f918
      Omid Azizi authored
      
      Summary:
      The custom decoder is faster by ~7%
      
      ```
      ----------------------------------------------------------------
      Benchmark                      Time             CPU   Iterations
      ----------------------------------------------------------------
      BM_custom_body_parser      58485 ns        58480 ns        10275
      BM_pico_body_parser        62858 ns        62841 ns        11012
      ```
      
      Test Plan: There's already test coverage for this. Just flipping the switch.
      
      Reviewers: #stirling, jamesbartlett, yzhao
      
      Reviewed By: #stirling, yzhao
      Signed-off-by: default avatarOmid Azizi <oazizi@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10825
      
      GitOrigin-RevId: 97694b23e8f9a3f7dc9a8bba7e6e10122d95d584
      4d96f918
    • Omid Azizi's avatar
      NetworkStatsConnector Bug-fix: More consistent stats when container PIDs switch namespaces · e56bd345
      Omid Azizi authored
      
      Summary:
      In the rare case when a container process switches namespaces with a call to setns,
      NetworkStatsConnector can incorrectly produce stats from the wrong namespace.
      
      Mitigate this effect by choosing the oldest PID in the container. This provides stability in which PID is chosen for producing stats.
      
      We also assume here that the oldest PID is not the one switching namespaces, so a more robust fix may be required in the future.
      
      Note that this bug was found by monitoring the PEM itself. Stirling forks a Java agent attach process which then changes namespaces,
      causing mis-reported network stats.
      
      Test Plan: Test added for the ordering. Also checked manually that the mis-reported network stats are fixed.
      
      Reviewers: #stirling, michelle, jps
      
      Reviewed By: #stirling, jps
      Signed-off-by: default avatarOmid Azizi <oazizi@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10823
      
      GitOrigin-RevId: 021a2d1d77805a7713a4e9a2acfae35c57a14255
      e56bd345
    • Omid Azizi's avatar
      Add TestContext for limiting Stirling context to test PIDs. · 663d6d7d
      Omid Azizi authored
      
      Summary: This is a test utility for limiting Stirling's monitoring to a smaller set of PIDs.
      
      Test Plan: This is test tooling, so rely on existing tests.
      
      Reviewers: #stirling, jps
      
      Reviewed By: #stirling, jps
      
      Subscribers: jps
      Signed-off-by: default avatarOmid Azizi <oazizi@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10827
      
      GitOrigin-RevId: 47641e1848126bedfd6b8ce9b20862f22fe9f556
      663d6d7d
  4. 21 Feb, 2022 1 commit
  5. 18 Feb, 2022 7 commits
    • James Bartlett's avatar
      [PP-3211] Add mysql data to socket tracer benchmark. · 0d7d2c39
      James Bartlett authored
      
      Summary:
      Adds a `RecordGenerator` for mysql data.
      
      Blocked by D10798
      
      Test Plan:
      No production changes.
      Tested that all the input mysql records get output by the benchmark.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      JIRA Issues: PP-3211
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10799
      
      GitOrigin-RevId: c7042be8d4ea007ee3318a77f77976ca344087bd
      0d7d2c39
    • Pete Stevenson's avatar
      PerfProfiler: change args to java agent to indicate the symbol file path (vs.... · 74dc7d9e
      Pete Stevenson authored
      PerfProfiler: change args to java agent to indicate the symbol file path (vs. name stem, which was awkward).
      
      Summary: Previously, the symbolization agent args indicated a file name stem because the symbol file name was uniquified based on target process pid. The name was given as a name stem (in case the agent needed to create a different file name for the txt log file). This setup was awkward both because it was a name stem and because it required coordination of the symbol file name in several places. Now we uniquify based on injecting the target process pid into the artifacts path. In this diff, we change the JVMTI symbolization agent args to indicate only the artifacts path and place a standard symbol file name in the shared header file.
      
      Test Plan: Existing.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      Subscribers: oazizi
      
      JIRA Issues: PP-2982
      Signed-off-by: default avatarPete Stevenson <jps@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10818
      
      GitOrigin-RevId: 902ec31bb1f306ded30f5e9714801c306207506c
      74dc7d9e
    • James Bartlett's avatar
      DataStreamBuffer: move the implementation of DataStreamBuffer behind a private... · 7389a1c6
      James Bartlett authored
      DataStreamBuffer: move the implementation of DataStreamBuffer behind a private implementation class.
      
      Summary: This diff allows for a runtime flag to switch which DataStreamBuffer implementation to use. Currently, there's only the one implementation, but a future diff will add a new DataStreamBuffer implementation.
      
      Test Plan:
      This diff is mostly moving code around, but to be safe I tested:
      - Existing data stream buffer tests pass
      - Skaffold deploy works, and data can be seen as usual.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      Subscribers: oazizi, yzhao
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10802
      
      GitOrigin-RevId: 54f35a4df4b04d3dbe6225458fb1d1e9beff39c2
      7389a1c6
    • James Bartlett's avatar
      [PP-3211] Add postgres data to socket tracer benchmark. · 42d7d7c9
      James Bartlett authored
      
      Summary:
      Adds a `RecordGenerator` that generates postgres data.
      
      Blocked by D10797
      
      Test Plan: No production changes. Tested that the benchmark outputs all the input records for the new postgres data.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      JIRA Issues: PP-3211
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10798
      
      GitOrigin-RevId: 5594ea97de0e8815af417d09fe1f8386b6981271
      42d7d7c9
    • James Bartlett's avatar
      [PP-3211] Add support for "gappy" data in socket tracer benchmark. · daaa1a2f
      James Bartlett authored
      
      Summary:
      Adds two kinds of `PosGenerator`'s that generate "gappy" data (i.e. simulate data from BPF that has large gaps in the data stream):
      - One generates data that has gaps between simulated polling iterations.
      - One generates data that has gaps within a single polling iteration.
      
      Blocked by D10793
      
      Test Plan: No production changes. Tested that the new PosGenerator's work with the benchmark.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      Subscribers: oazizi
      
      JIRA Issues: PP-3211
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10797
      
      GitOrigin-RevId: 5f85395d1e06c7843e50836903a7075b56645f93
      daaa1a2f
    • Pete Stevenson's avatar
      PerfProfiler: remove an unnecessary include. · 2547cf9c
      Pete Stevenson authored
      
      Summary: We included <utility> based on a complaint from our linter, but the code subsequently changed leaving a stale include. Here, we remove that.
      
      Test Plan: Build process.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      JIRA Issues: PP-2982
      Signed-off-by: default avatarPete Stevenson <jps@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10811
      
      GitOrigin-RevId: 8d74f0747abe2daea37e634db28863ae25837d21
      2547cf9c
    • James Bartlett's avatar
      DataStream: Move DataStreamBuffer::ShrinkToFit() into CleanupEvents() · 72381026
      James Bartlett authored
      
      Summary:
      When we moved the retention size limit restriction from DataStream::ProcessBytesToFrames into CleanupEvents, we neglected to also move the ShrinkToFit so that the DSB would be shrunk to the retention size.
      This diff just moves the ShrinkToFit into CleanupEvents.
      No need to still call ShrinkToFit in ProcessBytesToFrames because CleanupEvents will be called briefly after the end of ProcessBytesToFrames no matter what.
      
      Test Plan:
      Existing tests pass.
      Tried it out in skaffold and made sure scripts work as normal.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10813
      
      GitOrigin-RevId: ef4a1e4aa97f56573d9b53b3bcc6530617a0f8dd
      72381026
  6. 17 Feb, 2022 11 commits
    • Phillip Kuznetsov's avatar
      [Cleanup] Remove unused variable · bea3b23b
      Phillip Kuznetsov authored
      
      Summary: Simple fix. Something I spotted while documenting the data pipelines.
      
      Test Plan: Unit tests pass
      
      Reviewers: michelle, nserrino
      
      Reviewed By: michelle
      Signed-off-by: default avatarPhillip Kuznetsov <pkuznetsov@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10814
      
      GitOrigin-RevId: b33a01cf0024d2dc21b02e71d9f10a02eb7d16ab
      bea3b23b
    • Omid Azizi's avatar
      Mux Tracer: ByteSize · 63394904
      Omid Azizi authored
      
      Summary: Update the Mux protocol ByteSize() function to be more accurate.
      
      Test Plan: TBD.
      
      Reviewers: #stirling, rcheng
      
      Reviewed By: #stirling, rcheng
      
      Subscribers: rcheng
      Signed-off-by: default avatarOmid Azizi <oazizi@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10745
      
      GitOrigin-RevId: 0ae374ae50b05e71b5c3a6b3dbca5b042e4a33b9
      63394904
    • Omid Azizi's avatar
      Naming: Rename test container · bbb9715e
      Omid Azizi authored
      
      Summary: Use more inclusive language.
      
      Test Plan: This is just a name change. Rely on existing tests.
      
      Reviewers: #stirling, yzhao
      
      Reviewed By: #stirling, yzhao
      Signed-off-by: default avatarOmid Azizi <oazizi@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10808
      
      GitOrigin-RevId: 9f3c06a144a838e09347adf6528eba8da49e9387
      bbb9715e
    • James Bartlett's avatar
      [PP-3211] Add socket tracer benchmark framework. · 403a3edd
      James Bartlett authored
      
      Summary:
        - Added a data generation framework, that separates the generation of the data from the generation of the `pos` byte label for each event. Users of the data generation framework specify a `RecordGenerator` and a `PosGenerator`. The `RecordGenerator` has a single method `Next` that should return a record (a collection of traffic direction, frame pairs) that represents a single record that would be pushed to the TableStore if successfully parsed. The `PosGenerator` has a method that returns the byte position for the next event given the next event's size, and a method to signal the next polling iteration of data (useful for adding a gap in byte position between simulated polling iteration). Using the provided `PosGenerator` and `RecordGenerator` the data generation framework creates events per connection per simulated polling iteration (RecordGenerators are instantiated per connection and PosGenerators are instantiated per direction per connection).
        - To keep this diff as small as possible I only added a minimal set of Pos and Record generators in order to benchmark only HTTP1 traffic with continuous `pos` labels. Future diffs add more interesting PosGenerators as well as simple RecordGenerators for a variety of protocols.
        - Added the benchmark itself. The benchmark first generates all the necessary data using the above framework (as specified by `BenchmarkDataGenerationSpec`). Then during a single benchmark iteration, it will for each simulated "polling" iteration: push each event for that iteration to the SocketTraceConnector as if it had come from BPF (eg. via HandleDataEvent), then call `TransferData` once. During the first benchmark iteration the `MemoryTracker` will be enabled to keep track of starting, ending and peak memory usage for that particular benchmark. Finally after all benchmark iterations, the benchmark reports a wide variety of statistics. The statistics reported are: `PollIters` (the number of simulated polling iterations), `AllocPeak` (the maximum memory used during the benchmark minus the memory used before the benchmark started processing data), `PhysMem{Start,End}` (The physical memory used at the start/end of the benchmark, physical memory includes memory that the program has freed to the allocator but the allocator hasn't released to the OS), `AllocMem{Start,End}` (The memory that has not been freed to the allocator at the start/end of the benchmark), `EquivalendWorkloadThroughput` (The throughput in B/s of a workload that would produce similar load to this benchmark), `Bytes{Input/Output}`/`Records{Input/Output}` (The amount of bytes/records of data input/output to/from the benchmark, note that records input should equal records output if no data was lost), and finally `NumEvents` (the total number of events processed).
        - Because there are so many statistics, I added a variety of flags to control what statistics get printed, by default only `AllocPeak` and `PollIters` are displayed.
      
      Blocked by D10786, D10784, D10750
      
      Test Plan: No production changes, tested that the new benchmarking code runs the benchmark successfully, and that for each protocol the generated data can be successfully parsed.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      Subscribers: yzhao, oazizi
      
      JIRA Issues: PP-3211
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10793
      
      GitOrigin-RevId: b0e62c9dbdb3b9e6f70691d127bf74b1b207a78e
      403a3edd
    • Natalie Serrino's avatar
      Add semantic type rules for px.abs · bc676621
      Natalie Serrino authored
      
      Summary: I tried to do a px.abs on the result of px.parse_duration, but noticed that it didn't preserve the DURATION_NS type. This diff adds the correct semantic type to the result of px.abs.
      
      Test Plan: not needed, we use these same rules elsewhere, just need to add them here
      
      Reviewers: philkuz, jamesbartlett
      
      Reviewed By: jamesbartlett
      Signed-off-by: default avatarNatalie Serrino <nserrino@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10809
      
      GitOrigin-RevId: d7a22eac106a04fa7ac9c806457b594f18b1887d
      bc676621
    • yzhao1012's avatar
      [CLEANUP] Move --sources flag to stirling.cc so it takes effect for PEM as well · ca0bae76
      yzhao1012 authored
      
      Summary: This allows skaffold to change PEM source connectors without making code changes
      
      Test Plan: Tested with skaffold
      
      Reviewers: #stirling, jps
      
      Reviewed By: #stirling, jps
      
      Subscribers: oazizi
      Signed-off-by: default avataryzhao1012 <yzhao@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10794
      
      GitOrigin-RevId: 637ba96e276b1a5c3edecd1d16a4e36569fb9400
      ca0bae76
    • yzhao1012's avatar
      [CLEANUP] Move system_info.{cc,h} to src/stirling/utils · 2458fbb0
      yzhao1012 authored
      
      Summary:
      One example of improving organization. Similar headers, like system_headers.{cc,h} are already
      in src/stirling/utils.
      
      Test Plan: Move code files, Jenkins build covers this
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      Signed-off-by: default avataryzhao1012 <yzhao@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10806
      
      GitOrigin-RevId: 3f869ba454c22b363f428ea7614bfd17cd6246f2
      2458fbb0
    • James Bartlett's avatar
      [PP-3211] Add cql wire data generation. · 2dc928ce
      James Bartlett authored
      
      Summary:
      For the purposes of the benchmark, we want to be able to generate a bunch of wire data for each protocol. This diff adds the utilities to make data generation easier. These utilities could also be used in the tests instead of the ad-hoc char[] byte buffers that the tests currently used. This diff adds a couple examples in the tests to demonstrate this usage.
      
      Note, this diff also restructures the build file so that test utilities don't get built into the main binary.
      
      Test Plan: No functional changes, so relying on existing tests. This diff also increases test coverage.
      
      Reviewers: #stirling, oazizi
      
      Reviewed By: #stirling, oazizi
      
      Subscribers: oazizi, yzhao
      
      JIRA Issues: PP-3211
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10784
      
      GitOrigin-RevId: 9317f1562fc37ce1a7155599e7f8d493456a20d2
      2dc928ce
    • Omid Azizi's avatar
      DNS parser ByteSize(): Re-implement to avoid const_cast · 4fe6d481
      Omid Azizi authored
      
      Summary:
      The old implementation of DNS' ByteSize() used a const_cast which can be avoided.
      
      This diff cleans that up.
      
      Test Plan: No functional change, so existing tests.
      
      Reviewers: #stirling, yzhao
      
      Reviewed By: #stirling, yzhao
      
      Subscribers: yzhao
      Signed-off-by: default avatarOmid Azizi <oazizi@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10776
      
      GitOrigin-RevId: 802c504f386745585438be885b9888ab32d35edc
      4fe6d481
    • James Bartlett's avatar
      [PP-3211] Add pgsql wire data generation. · 45433dce
      James Bartlett authored
      
      Summary:
      For the purposes of the socket tracer benchmark, we want to be able to generate a bunch of wire data for each protocol. This diff adds the utilities to make data generation easier for pgsql. These utilities could also be used in the tests instead of the ad-hoc char[] byte buffers that the tests currently used. I added a couple examples in the tests to demonstrate this usage.
      
      This diff also restructures the build file, such that the test data is not compiled with the main cc library.
      
      Test Plan: No functional changes, relying on existing tests. Note that this diff increases test coverage.
      
      Reviewers: #stirling, yzhao
      
      Reviewed By: #stirling, yzhao
      
      Subscribers: yzhao
      
      JIRA Issues: PP-3211
      Signed-off-by: default avatarJames Bartlett <jamesbartlett@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10786
      
      GitOrigin-RevId: 6d379b8aac9964bc27959f0378483d29d911409b
      45433dce
    • Vihang Mehta's avatar
      Convert to POST message and parse params from JSON body · 819e5959
      Vihang Mehta authored
      Summary: Use POST message type and JSON parse params.
      
      Test Plan: `curl -X POST -vvv --raw  -H 'Content-Type: application/json' --data '{"chunk_size":10, "body_size":100}' http://localhost:8111/`
      
      
      
      Reviewers: zasgar, nserrino
      
      Reviewed By: zasgar
      Signed-off-by: default avatarVihang Mehta <vihang@pixielabs.ai>
      
      Differential Revision: https://phab.corp.pixielabs.ai/D10804
      
      GitOrigin-RevId: fea928b276a065840723681a097042a5696a11a5
      819e5959