With Go modules, `go mod tidy` supplants `vendorfmt`. Unfortunately,
`tidy` will try to reach out to the network and download modules, and
there is no way to disable that behavior (e.g. the -mod=vendor) option
does not apply. This means we cannot use the `tidy` target in nomad
enterprise, which will be unable to reach private repositories like
consul-enterprise.

This isn't a big deal, since `vendorfmt` served the purpose of rewriting
the output of `govendor`, wheras `tidy` is a part of the `sync` target
that is required to be run when modifying dependencies anyway.

go-bindata released tags are very old, few years old, so ensure we use
the latest master as of now.

Point to gopsutil fork with shirou/gopsutil#889 changes. I used hashicorp/gopsutil rather than pointing to https://github.com/Lomanic/gopsutil/tree/issue832 directly just in case..

This PR switches the Nomad repository from using govendor to Go modules
for managing dependencies. Aspects of the Nomad workflow remain pretty
much the same. The usual Makefile targets should continue to work as
they always did. The API submodule simply defers to the parent Nomad
version on the repository, keeping the semantics of API versioning that
currently exists.

We have been using fatih/hclfmt which is long abandoned. Instead, switch
to HashiCorp's own hclfmt implementation. There are some trivial changes in
behavior around whitespace.

Use v1.1.5 of go-msgpack/codec/codecgen, so go-msgpack codecgen matches
the library version.

We branched off earlier to pick up
https://github.com/hashicorp/go-msgpack/commit/f51b5189210768cf0d476580cf287620374d4f02
, but apparently that's not needed as we could customize the package via
`-c` argument.

Examples for HTTP based task-group service healthchecks are
covered by the `countdash` demo, but gRPC checks currently
have no runnable examples.

This PR adds a trivial gRPC enabled application that provides
a Service implementing the standard gRPC healthcheck interface.

Running `make dev` runs `hclfmt`, but this isn't checked as part of
CI. That makes it possible to merge un-formatted HCL and Nomad
jobspecs that later will make for dirty git staging areas when
developers pull master.

This changeset adds HCL linting to the `make check` target.

Use go mod for github.com/hashicorp/go-bindata/go-bindata and
github.com/elazarl/go-bindata-assetfs/go-bindata-assetfs but use
`@master` to pull the latest master.  These packages don't have release
tags so `@master` worksaround it.

Adding `-trimpath` to builds removes the local working directory from
the goroutine stack traces, which makes our builds more reproducible
and doesn't leak information about our local development workstations
or CI environment.

These dep tools aren't tagged properly so shouldn't be installed with go
mod

This allows using https download and go mod cache proxies, over using
git and downloading entire dependencies git history, hopefully,
resulting into a faster installation process.

IBM testing.

This check fails the lint-go CI job if nomad references a package that
is not vendored.

Ran into this missing target on ppc64le.  Consul had no issue.

Emit a slightly helpful message when /api depends on nomad internal
packages.

Modernize Vault integration/e2e test a bit:

- Download from releases.hashicorp.com instead of using a hardcoded list
- Remove old unused make target e2e-test
- Use NOMAD_E2E env var instead of -integration flag
- Add a README

On my machine with ~250 Mbps internet it takes ~400s to download all
Vault binaries.

You'd think since golangci-lint embeds misspell we could use that,
but it fails to run if it finds no Go source files, which is the
case in our website/ directory that we want to check.

gometalinter has been deprecated, with golangci-lint as its spiritual
and recommended successor. Here we switch to using it with an equivalent
configuration, albeit with newer versions of some linters.

To maintain compatibility with existing settings, we have a couple of
things disabled here, specifically:

- tests
        We have a lot of unused code in our tests that choke deadcode.
        We should attempt to clean these up soon so that we can lint our
        testcode.
- govet.check-shadowing = false
        This breaks on redefining `err` which we do all over the nomad
        codebase.

Co-Authored-By: Danielle <dani@hashicorp.com>

This is an attempt to ease dependency management for external driver
plugins, by avoiding requiring them to compile ugorji/go generated
files.  Plugin developers reported some pain with the brittleness of
ugorji/go dependency in particular, specially when using go mod, the
default go mod manager in golang 1.13.

Context
--------

Nomad uses msgpack to persist and serialize internal structs, using
ugorji/go library.  As an optimization, we use ugorji/go code generation
to speedup process and aovid the relection-based slow path.

We commit these generated files in repository when we cut and tag the
release to ease reproducability and debugging old releases.  Thus,
downstream projects that depend on release tag, indirectly depends on
ugorji/go generated code.

Sadly, the generated code is brittle and specific to the version of
ugorji/go being used.  When go mod picks another version of ugorji/go
then nomad (go mod by default uses release according to semver),
d...

This fixes a bug where some packages tests run twice in CI, because they
are subpackages of ones with dedicated jobs.  For example, notice
`client/allocrunner` tests ran in `test-client`[1] and `test-other`[2] despite
them being in the same workflow[3].

[1] https://circleci.com/gh/hashicorp/nomad/5262
[2] https://circleci.com/gh/hashicorp/nomad/5261
[3] https://circleci.com/workflow-run/ee7a9167-8fc7-49a5-897c-6c967e966014

We currently use an container image for `test-devices` job only; while
all other jobs use machine executor.

This allows us to switch golang and protoc verions easily without
manually managing Docker images (which requires building them manually
on a dev machines, etc).  All that while, we install dependencies on
every build in all other jobs..

`test-devices` now is one of the fastest jobs and isn't a constraint or
a bottleneck, so increasing its overhead by few seconds doesn't hurt the
overall developer iteration.

If we split tests effectively later, we can revisit.

Fixes `test-devices` job

Allow users to customize their makefiles by adding custom
targets/variables without checking them in.