• Natalie Serrino's avatar
    PP-2117: Add the concept of an 'external' GRPC Sink to the compiler · ad749443
    Natalie Serrino authored
    Summary:
    In the new end-to-end streaming Vizier, GRPC sinks will stream results directly to the query broker, rather than Kelvin buffering up the final results and sending it in batch to the query broker. That means there are two 'types' of GRPC sinks in that system: internal GRPC sinks which send mid-query, intermediate data to GRPC sources on another Carnot instance, and external GRPC sinks which send complete results to the query broker or another external address. In the internal GRPC Sink case, the node only needs to know the destination ID of the GRPC Source node that it's sending the data to. In the external GRPC Sink case, the node needs to know the name and schema of the output table.
    
    In this diff, the concept of the external GRPC sink is introduced. Changes to rules are made so that things like automatically adding a limit to memory sinks will also apply to these external GRPC sinks. Next diff, the compiler will change so that px.display automatically results in these external GRPC sinks rather than memory sinks.
    
    Test Plan: added / existing
    
    Reviewers: philkuz, zasgar, #engineering
    
    Reviewed By: philkuz, #engineering
    
    JIRA Issues: PP-2117
    
    Differential Revision: https://phab.corp.pixielabs.ai/D5966
    
    GitOrigin-RevId: 472d7f33454b3b1df3ee5183e80cf62966de1d6f
    ad749443