Previously, Rook Agent and Discovery DaemonSet deployment didn't allow
adding nodeAffinity. This commit adds nodeAffinity spec to daemonSet
deployment, which can be configured through environment variables in
operator deployment yaml.

+ Support multiple LabelKey, each with multiple LabelValue
+ Support multiple LabelKey with no value
Signed-off-by: Umanga Chapagain <chapagainumanga@gmail.com>

Signed-off-by: travisn <tnielsen@redhat.com>

OwnerReferences are now created with the fully qualified apiVersion
such that the references will work properly on OpenShift.
Signed-off-by: travisn <tnielsen@redhat.com>

This requirement is long gone and was reported in 2017. Clients should
have been updated by then, so there is no reason to set the CRUSH
tunable to such an old client. Actually, we should let Ceph run its own
tunable. They can always be changed later. Also, this was really
restrictive and applied on every orchestration and thus would override
any other config done by an administrator. Even worse, this will trigger
data movement back and forth...

Closes: https://github.com/rook/rook/issues/3138

Signed-off-by: Sébastien Han <seb@redhat.com>

When the operator first starts, the only operation needed
is to watch for new cephcluster crds to be created and
start the discovery to find available devices. The flexvolume
agent, the csi driver, and the volume provisioning can all be delayed
starting until the first cluster is created.
Signed-off-by: travisn <tnielsen@redhat.com>

This commit does multiple things:

* remove support for AllNodes where we would deploy one rgw per node on
all the nodes.
* a transition path is implemented in the code so that if someone has an
existing deployment, daemonsets will be removed and replaced by an
deployments.
* when using "instances", each rgw deployed has its own key which makes
Ceph reporting the exact number of rgw running, see:

```
[root@rook-ceph-operator-775cf575c5-bh4sr /]# ceph -s
  cluster:
    id:     611fcf39-0669-4864-9a12-debb35c0397a
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 12h)
    mgr: a(active, since 12h)
    osd: 3 osds: 3 up (since 12h), 3 in (since 12h)
    rgw: 3 daemons active (my.store.a, my.store.b, my.store.c)

  data:
    pools:   6 pools, 600 pgs
    objects: 235 objects, 3.8 KiB
    usage:   3.0 GiB used, 84 GiB / 87 GiB avail
    pgs:     600 active+clean
```

Closes: https://github.com/rook/rook/issues/2474, https://github.com/rook/rook/issues/2957 and https://github.com/rook/rook/issues/3245

Signed-off-by: Sébastien Han <seb@redhat.com>

When OSDs are running on Nautilus we always disable old osd features and
aplpy the onces for Nautilus as described in the upgrade doc.
During an upgrade or the next time an orchestration will be called the
command will be applied. The command is idempotent so we can run it each
time.
This can be backported for 1.0.3

Closes: https://github.com/rook/rook/issues/2960

Signed-off-by: Sébastien Han <seb@redhat.com>

As per: ceph/ceph#26599, Beast is now the
default fronted for rados gateway.
Newly created cluster as of Nautilus will use it by default.

Re-added version of 03587352


Resolves: #2707
Signed-off-by: Sébastien Han <seb@redhat.com>

We now check if the pod responds on http port 80.
Signed-off-by: Sébastien Han <seb@redhat.com>

- Updated code to use deviceClass property when a pool is created for both "replicated" and "erasure code".
- Updated "ceph crush rule create-..." command to use "create-replicated" instead of "create-simple"
- Updated unit tests
- Updated (ceph-pool-crd.md) documentation to reflect the changes.
- Updated pending release notes.
Signed-off-by: Santosh Pillai <sapillai@redhat.com>

Signed-off-by: Jose A. Rivera <jarrpa@redhat.com>

Signed-off-by: travisn <tnielsen@redhat.com>

The flex settings cannot be passed to the driver with environment variables.
There is no context available for returning the flex settings except
that the flex driver will look in a config file in the same directory.
These settings must be valid or else the driver will return the default
settings.
Signed-off-by: travisn <tnielsen@redhat.com>

The alpha, beta, and stable channels do not match the rook release process.
Each storage provider defines their own stability based on the CRDs
rather than defining it with the release process. Rook really
only has two release streams: master and the official releases.
Master is published with each merge, while releases go through
a signoff process to ensure quality. Thus, the alpha, beta,
and stable channels are removed and replaced with a
single release channel.
Signed-off-by: travisn <tnielsen@redhat.com>

This seems to fix issues with Minio which have just now appeared out of
nowhere.
Signed-off-by: Alexander Trost <galexrt@googlemail.com>

This reverts commit b7a564e9, reversing
changes made to 6996571b

.

Reverting: Linking ceph noout to a node taint.
The taint is applied for reasons other than maintenance as well.
This also needs a way to respect user-set noout.
Need to revisit this with a better design.
Signed-off-by: Rohan CJ <rohantmp@gmail.com>

The CephCluster CR contains settings that are needed by other
CRs to configure the Ceph daemons. When the CephCluster CR
is updated, the updates will now be passed on to each of the
CR controllers to ensure the daemons are updated properly
without requiring an operator restart.

When calling the controllers from another controller,
we ensure that only a single goroutine is handling
CRs at any given time to prevent contention across
multiple CRs of the same type.
Signed-off-by: travisn <tnielsen@redhat.com>

1. Add support of `metadataDevice` & `databaseSizeMB` options to
   OSDs provisioned by `ceph-volume`.

2. Add `ceph-volume --report` output before the actual run to
   provide more info.
Signed-off-by: Ash Wu <hSATAC@gmail.com>
Co-authored-by: Ash Wu <hSATAC@gmail.com>
Co-authored-by: Chia-liang Kao <clkao@clkao.org>

Signed-off-by: travisn <tnielsen@redhat.com>

Add a 'ceph-version' label to application controllers in the same manner
as the prior 'rook-version' label to help with upgrades.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>

- Add noout functions to osd client
- export the functions DiscoverStorageNodes,GetAllStorageNodes
  from pkg/operator/ceph/cluster/osd
- Update PendingReleaseNotes.md
Signed-off-by: Rohan CJ <rohantmp@gmail.com>

this patch monitors udev events from the block subsystem via the udevadm
tool. it watches for add and delete events within a specified period
(e.g. 2 seconds) and then emits a trigger which starts a new device
probe operation.
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

We now run all the daemon with a new option from Nautilus 14.2.1 which
allows us to tell to a daemon to not log on file. However, we can decide
to activate logging by editing the configuration flag 'log_to_file' via
the centralized config store like this for a particular daemon:

ceph config set mon.a log_to_file true

This is useful when a daemon keeps crashing and we want to collect log
files on the system.

Fixes: https://github.com/rook/rook/issues/2881

Signed-off-by: Sébastien Han <seb@redhat.com>

Make the Ceph operator more cautious about when it decides to remove
nodes from the Rook-Ceph cluster which are acting as osd hosts.

When `useAllNodes` is set to `true` we assume that the user wants to
have the most hands-off experience. Node removals are allowed when a
node is delted from Kubernetes and when a node has its taints/affinities
modified by the user (but not by automatic k8s modification as much as
possible).

When `useAllnodes` is set to `false` the only time a node is removed is
if it is removed from the Ceph cluster definition.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>

The CephCluster resource will now expose the health of the ceph cluster
so admins can query the status with k8s api or kubectl instead of
needing to run the rook toolbox. The operator will query the
ceph status periodically and update the status in the custom resource.
Signed-off-by: travisn <tnielsen@redhat.com>

Signed-off-by: travisn <tnielsen@redhat.com>

Signed-off-by: travisn <tnielsen@redhat.com>

The desired number of mons could change depending on the number of nodes in a cluster.
For example, three mons would be the min number of mons in a production cluster with at
least three nodes. If there are five or more nodes, the number of mons could increase
to five in order to increase the failure tolerance to two nodes.

This is accomplished by a new setting in the cluster CRD preferredCount. If the number
of hosts exceeds preferredCount, more mons are added to quorum. If the number
of hosts drops below the preferred count, the operator would reduce the quorum size to
the smaller desired count.
Signed-off-by: travisn <tnielsen@redhat.com>

Signed-off-by: Stefan Haas <shaas@suse.com>

In the form of mon, mgr, mds, and rgw, convert the rbd-mirror to be
configured completely from the operator. This is a straightforward
conversion with one functional addition: the rbd-mirror daemon stores
*no* data and has no default data dir, so the concept of a `Dataless`
daemon is here introduced.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>

Configure the Ceph rgw daemon completely from the operator a la the
recent changes to the Ceph mon, mgr, and mds operators.

Create the rgw deployment or daemonset first, and then create the
keyring secret for the object store with its owner reference as the
corresponding deployment or daemonset. When the replication controller
is deleted, the secret is also deleted.

The RGW's mime.types file is now stored in a configmap with a different
file created for each object store. This is primarily just a means to
get the mime.types file into the rgw pod, but the added benefit is that
the administrator can modify the configmap, which could reduce
susceptibility to file type execution vulnerabilities (worst case).
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>

Configure the Ceph mds daemon completely from the operator a la the
recent changes to the Ceph mon and mgr operators.

Create the mds deployments first and then
create the keyring secrets for them with their owner reference as the
corresponding deployment. This will mean that the secrets do not need to
be micromanaged. When the deployment is deleted, the secret is also
deleted. This has not been necessary for the mons or the manager since
the mons share a keyring with a lifespan of the cluster, as does the
mgr, which currently has single-mgr support only.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>

Signed-off-by: Huamin Chen <hchen@redhat.com>

Signed-off-by: Huamin Chen <hchen@redhat.com>

Configure the Ceph mgr daemon completely from the operator a la the
recent changes to the Ceph mon operator.

The pod spec for the mgr changed quite a bit, and instead of updating
the unit tests of questionable use, some additional unit test tools
applicable to any Ceph daemon have been added and used with the mgr. The
mgrs unit tests should now be more useful and get in the way of devs
less, and they can be used by other daemons later.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>

Make the Rook config-init unnecessary for mons, and remove that init
container. Perform all mon configuration steps in the operator, and set
up the mon pods and k8s environment such that only Ceph containers are
needed for running mons.

This should help streamline changes to the mons, as there will be no
need to change the `daemon/mon` code or `cmd/rook/ceph` code with mon
changes in the future.

This work starts to lay the groundwork for supporting the
`design/ceph-config-updates.md` design.

Notable new bits:

Create a keyring secret store helper for storing dameon keyrings, and
use it to store the mon keyring. Mon pods mount the keyring into a
k8s secret-backed volume.

Create a configmap store for the Ceph config file which can be mounted
into pods/containers directly to /etc/ceph/ceph.conf. Also store
individual mon_host and mon_initial_members values which can be mapped
into pods as environment variables and used in Ceph commandline flags,
enabling the mon pods to have the most up-to-date information about the
mon cluster when restarting and without need for operator intervention.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>

We now refuse to start more than one monitor on the same machine if
hostnetworking is enabled.
Supporting this creates a lot more complexity from the rook side. Also
having 3 monitors running on the same machine is not a production setup.

This means multi-cluster support on the same machine
is not possible anymore when hostnetworking is enabled.

Resolves: https://github.com/rook/rook/issues/2604

Signed-off-by: Sébastien Han <seb@redhat.com>

Rook's behavior should not be to create a default OSD in dataDirHostPath
when no devices are present on a node. Any preexisting fallback osd
should be kept as long as no dirs or disks have been specified to keep
the legacy behavior for those clusters running with these osds in place.
The legacy deletion behavior is also kept, removing the osd as soon as
any dir or device is specified other than the dataDirHostPath dir.
Signed-off-by: Blaine Gardner <blaine.gardner@suse.com>

Signed-off-by: travisn <tnielsen@redhat.com>

Co-authored-by: jtlayton <jlayton@redhat.com>
Signed-off-by: travisn <tnielsen@redhat.com>