Manual deployment without Ceph

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Manual deployment without Ceph

rdowell
I've been working with John Calcote who's posted several threads here in the last couple weeks.  Now that we've more or less figured out what we wanted to do with regards to extending the VSM dashboard to support our hardware configuration we're trying to figure out how to make it mesh with our deployment and configuration scheme.

I've got a couple questions regarding VSM deployment and topology requirements as we try to figure out how best to deploy VSM along with the particular Ceph setup and other software that is needed for managing our hardware setup

1) The hardware setup that we're deploying (and where we've already been running Ceph by itself without VSM) has 4 storage and 3 monitor nodes, which are all discrete, so the monitors are not running on the same nodes as the storage.  Ideally, we'd like to also use one of the monitor nodes to run the VSM controller instead of having to add an additional node to the setup.  Is it possible to do this in VSM auto-deployment?  I.E, Can I setup the manifest for auto-deploy as follows and have it deploy successfully:
manifest/
    node1/
        cluster.manifest # Controller
        server.manifest # monitor role only
    node2/
        server.manifest # monitor role only
    node3/
        server.manifest # monitor role only
    node4/
        server.manifest # storage role only
    node5/
        server.manifest # storage role only
    node6/
        server.manifest # storage role only
    node7/
        server.manifest # storage role only

2) We're also deploying a custom version of Ceph, but I've seen from other discussions here that it should be possible to support that in VSM deployment by configuring our own apt repository so that our version is resolved before the distro-default one.  The problem is that from what I've seen of the VSM deployment, Ceph deployment and configuration is pretty tightly tied into the VSM deployment and thus it seems like it might not be possible to configure Ceph in the way we want to inside the VSM deployment.  Because of this, we would ideally be doing our Ceph deployment first and then deploying VSM afterward, but I've also seen it mentioned in many places that "VSM cannot manage a Ceph cluster it did not create" and that this is slated to be addressed in a future release.  I'm wondering if you can elaborate on this any more and whether that just means that the actual cluster creation step has to be done by VSM and we could still deploy the Ceph software to each node ourselves.  Also, is it possible to simply install each of the VSM packages to each node by hand rather than using the deployment scripts, so that we can use our Ceph deployment scripts that are already in place, install the VSM agent/API/dashboard/etc by hand, and then use the VSM GUI to create the Ceph cluster.

I realize the two scenarios I asked about are not compatible with each other, we're just trying to figure out what is and is not viable with regards to VSM installation so we can figure out the best way to use it in the setup we're trying to deploy.
Reply | Threaded
Open this post in threaded view
|

Re: Manual deployment without Ceph

rdowell
I think I answered my own question on part 1, as I did a test run of VSM auto-deploy successfully with one node designated as both the controller and a Ceph monitor, and successfully created the Ceph cluster after deployment succeeded
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

ywang19
Administrator
In reply to this post by rdowell

Hello,

 

You already got the answer for question 1), so I will answer the second one.

 

So far VSM can’t manage clusters not created by it, this is the current constraints. We are working on this, and have some groundwork ready. For your case, it’s actually not constraint to the feature. VSM allows a few approaches to adopt designated ceph version as described in FAQ section in INSTALL.md:

**Q: Can I define the ceph version I want to install?**

 

          A:

          VSM can work with different Ceph releases like Firefly, Giant and Hammer. By default, it will use the ceph version provided by OS distro, often it’s an latest stable version.

          If user expects to use some specific ceph version, he/she needs add the new ceph repo into system repository.

 

          For ubuntu, user could create /etc/apt/sources.list.d/ceph.list to override OS distro’s defaults before installation. For example, below commands could help setup a ceph repo for Hammer on ubuntu:

                   >

                   > sudo rm /etc/apt/sources.list.d/ceph.list

                   > echo deb http://ceph.com/debian-hammer/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

 

 

          For CentOS, creating /etc/yum.repos.d/ceph.repo at first, then filling the ceph.repo shown below:

                   ###################

                   [ceph]

                   name=Ceph packages for $basearch

                   baseurl=http://ceph.com/rpm-hammer/el6/$basearch

                   enabled=1

                   priority=2

                   gpgcheck=1

                   type=rpm-md

                   gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc

 

                   [ceph-noarch]

                   name=Ceph noarch packages

                   baseurl=http://ceph.com/rpm-hammer/el6/noarch

                   enabled=1

                   priority=2

                   gpgcheck=1

                   type=rpm-md

                   gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc

 

                   [ceph-source]

                   name=Ceph source packages

                   baseurl=http://ceph.com/rpm-hammer/el6/SRPMS

                   enabled=0

                   priority=2

                   gpgcheck=1

                   type=rpm-md

                   gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc

                   ##############

 

          In 2.0, VSM also provides ceph upgrade feature from Web UI under "Cluster Management" menu.

 

 

Generally, user could set ceph repositories at VSM deployment, or after deployed, user could access “cluster management\ceph upgrade” to refresh ceph packages to designated version.

 

At ceph deployment, VSM uses a default configuration. Considering user may change some settings later on, we provide a command line to allow user to customize ceph.conf after cluster is deployed.

 

Ø  Import_ceph_conf <cluster name> <new ceph.conf file>

 

 

 

-yaguang

 

From: rdowell [via vsm-discuss] [mailto:ml-node+[hidden email]]
Sent: Wednesday, September 30, 2015 5:08 AM
To: Wang, Yaguang
Subject: Manual deployment without Ceph

 

I've been working with John Calcote who's posted several threads here in the last couple weeks.  Now that we've more or less figured out what we wanted to do with regards to extending the VSM dashboard to support our hardware configuration we're trying to figure out how to make it mesh with our deployment and configuration scheme.

I've got a couple questions regarding VSM deployment and topology requirements as we try to figure out how best to deploy VSM along with the particular Ceph setup and other software that is needed for managing our hardware setup

1) The hardware setup that we're deploying (and where we've already been running Ceph by itself without VSM) has 4 storage and 3 monitor nodes, which are all discrete, so the monitors are not running on the same nodes as the storage.  Ideally, we'd like to also use one of the monitor nodes to run the VSM controller instead of having to add an additional node to the setup.  Is it possible to do this in VSM auto-deployment?  I.E, Can I setup the manifest for auto-deploy as follows and have it deploy successfully:
manifest/
    node1/
        cluster.manifest # Controller
        server.manifest # monitor role only
    node2/
        server.manifest # monitor role only
    node3/
        server.manifest # monitor role only
    node4/
        server.manifest # storage role only
    node5/
        server.manifest # storage role only
    node6/
        server.manifest # storage role only
    node7/
        server.manifest # storage role only

2) We're also deploying a custom version of Ceph, but I've seen from other discussions here that it should be possible to support that in VSM deployment by configuring our own apt repository so that our version is resolved before the distro-default one.  The problem is that from what I've seen of the VSM deployment, Ceph deployment and configuration is pretty tightly tied into the VSM deployment and thus it seems like it might not be possible to configure Ceph in the way we want to inside the VSM deployment.  Because of this, we would ideally be doing our Ceph deployment first and then deploying VSM afterward, but I've also seen it mentioned in many places that "VSM cannot manage a Ceph cluster it did not create" and that this is slated to be addressed in a future release.  I'm wondering if you can elaborate on this any more and whether that just means that the actual cluster creation step has to be done by VSM and we could still deploy the Ceph software to each node ourselves.  Also, is it possible to simply install each of the VSM packages to each node by hand rather than using the deployment scripts, so that we can use our Ceph deployment scripts that are already in place, install the VSM agent/API/dashboard/etc by hand, and then use the VSM GUI to create the Ceph cluster.

I realize the two scenarios I asked about are not compatible with each other, we're just trying to figure out what is and is not viable with regards to VSM installation so we can figure out the best way to use it in the setup we're trying to deploy.


If you reply to this email, your message will be added to the discussion below:

http://vsm-discuss.33411.n7.nabble.com/Manual-deployment-without-Ceph-tp161.html

To start a new topic under vsm-discuss, email [hidden email]
To unsubscribe from vsm-discuss, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
Hi Yaguang,

Thanks for the information.  Having a utility to deploy a modified ceph config seems like it might suit our needs well for our deployment scenario.  One issue I noticed though is that it doesn't appear to be part of the installation on Ubuntu:

cephuser@rdowell-vsm-controller:~$ dpkg -L vsm | grep import
/usr/local/lib/python2.7/dist-packages/vsm/openstack/common/importutils.pyc
/usr/local/lib/python2.7/dist-packages/vsm/openstack/common/importutils.py
/usr/local/lib/python2.7/dist-packages/vsm/openstack/common/importutils.pyo
cephuser@rdowell-vsm-controller:~$ dpkg -L vsm-deploy | grep import
cephuser@rdowell-vsm-controller:~$

I found the script in the source code, under both source/vsm/bin and suse/vsm/bin with minor differences, and it's also listed in the RPM package contents in source/vsm/vsm.spec and centos7/vsm/vsm.spec, but it appears it doesn't get included in the debian packages.  Is this intentional or is this a bug in the debian packaging?  For now I'll just copy the script directly from the source code to my controller system and continue working on other deployment issues.



I was also curious if you could elaborate a bit further regarding the requirement of deploying Ceph along with VSM.  For the time being, using VSM deployment while defining our own Ceph version is still looking like the best option, but our existing Ceph-only deployment also employs some third party software which I'm not yet sure is going to work with this deployment process.

Instead of using VSM auto-deployment, if I wanted to use our existing Ceph deployment, and then manually install vsm, vsm-dashboard, vsm-deploy packages on each system and then manually configure the VSM services, can you tell me whether this would work and if not, what are the limitations that prevent me from doing so?  We'd still use the VSM GUI to actually create the Ceph cluster after installing, but I'm trying to understand if the reason for the "VSM must create the cluster" requirement goes deeper than that

Thanks,

-Ryan
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

ywang19
Administrator
Ryan,

About the missing of the tool import_ceph_conf, it's a flaw when we clean up our installer script, and I caught it, and included it in 2.0 final version (build 216).

About to make vsm working with deployed ceph cluster, yes, it's a constraint so far for vsm. we already have some groundwork on it, expect to provide the functionality in next version, targeting in Q4.


-yaguang
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

jcalcote
rdowell wrote
I was also curious if you could elaborate a bit further regarding the requirement of deploying Ceph along with VSM.  For the time being, using VSM deployment while defining our own Ceph version is still looking like the best option, but our existing Ceph-only deployment also employs some third party software which I'm not yet sure is going to work with this deployment process.

Instead of using VSM auto-deployment, if I wanted to use our existing Ceph deployment, and then manually install vsm, vsm-dashboard, vsm-deploy packages on each system and then manually configure the VSM services, can you tell me whether this would work and if not, what are the limitations that prevent me from doing so?  We'd still use the VSM GUI to actually create the Ceph cluster after installing, but I'm trying to understand if the reason for the "VSM must create the cluster" requirement goes deeper than that...
Hi Yaguang,

This is very important to us. If you could provide some insight on side-by-side deployment, we could provide patches to your project that might save your team some effort. One way or the other, we will get this working, but it would be very nice for both of us if we could submit patches to you that help your project move forward also.

I guess what I'm asking for is that you allow us to contribute to VSM in a way that is helpful to us both. If you've laid groundwork for this effort then we should be able to find that groundwork near the tip of default in the VSM github repo. If you could provide a 1 or 2 paragraph overview of your architecture for this feature, with references to the groundwork code in the repo, then we could begin to work along side of you on this feature and submit those patches back to you. Since you're already considering this feature important and have plans to add it in Q4, we're only asking that you allow us to work on it for/with you.

In my mind, this is a better approach than forking VSM.

Thanks,
John
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
In reply to this post by ywang19
Earlier I found the Jira for the 'existing cluster' feature: https://01.org/jira/browse/VSM-53

I noticed in the description of the Jira that it sounds like the primary reason for not supporting this is concerns with clusters using a CRUSH map that does not work with the rules VSM expects to have by default:

One possible approach (still difficult, but tractable): Allow VSM to connect to an existing cluster if the CRUSH map is compatible with VSM - Support an option to identify preexisting Storage Group and Zone buckets in the cluster manifest file. This would likely require manual inspection of the existing CRUSH map, but might be possible to automate with a tool that parses an existing CRUSH maps and determine whether ti is VSM-compatible."
If I were able to get an example of the CRUSH map we're using, would it be possible for you to either:

A) Tell us whether our CRUSH map would be considered 'compatible' with VSM and direct us to how we could then deploy VSM on top of the cluster?

or

B) Tell us how we could use our CRUSH map instead of the defaults when deploying VSM, in which case we may be able to just use VSM deployment instead?
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

ywang19
Administrator

If you were able to share the CRUSHMAP, it certainly helps, better with ceph.conf together.

 

Current VSM uses below hierarchy in CRUSHMAP: Storage group -> zone -> host -> osd, one solution we are evaluating is to use nested zones to cover broader cases. If your CRUSHMAP could map into this hierarchy, it might be possible to be managed by VSM with below steps:

i)             Deploy VSM as usual, and fill correct storage group/osd mapping and zone information in server.manifest for each node

ii)           In VSM UI, “cluster management\create cluster”, there is a button named “integrate cluster” (it’s hidden in 2.0 final), you could make below modifications on controller node to enable it:

a.    Open /usr/share/vsm-dashboard/vsm_dashboard/dashboards/vsm/templates/vsm/flocking/_createcluster.html

b.    Remove “style=”display:none”:

  <input class="btn btn-primary integrate-cluster-commit" type="submit" style="display:none" value="{% trans "integrateCluster" %}" />

c.    Restart dashboard:

                                         i.    Sudo service apache2 restart

d.   Then after you clicked “create” button, on the new dialog, you will see “integrate cluster” button, which should help initialize vsm for existing cluster.

 

 

-yaguang

 

From: rdowell [via vsm-discuss] [mailto:ml-node+[hidden email]]
Sent: Wednesday, October 14, 2015 5:00 AM
To: Wang, Yaguang
Subject: RE: Manual deployment without Ceph

 

Earlier I found the Jira for the 'existing cluster' feature: https://01.org/jira/browse/VSM-53

I noticed in the description of the Jira that it sounds like the primary reason for not supporting this is concerns with clusters using a CRUSH map that does not work with the rules VSM expects to have by default:

One possible approach (still difficult, but tractable): Allow VSM to connect to an existing cluster if the CRUSH map is compatible with VSM - Support an option to identify preexisting Storage Group and Zone buckets in the cluster manifest file. This would likely require manual inspection of the existing CRUSH map, but might be possible to automate with a tool that parses an existing CRUSH maps and determine whether ti is VSM-compatible."

If I were able to get an example of the CRUSH map we're using, would it be possible for you to either:

A) Tell us whether our CRUSH map would be considered 'compatible' with VSM and direct us to how we could then deploy VSM on top of the cluster?

or

B) Tell us how we could use our CRUSH map instead of the defaults when deploying VSM, in which case we may be able to just use VSM deployment instead?


If you reply to this email, your message will be added to the discussion below:

http://vsm-discuss.33411.n7.nabble.com/Manual-deployment-without-Ceph-tp161p190.html

To start a new topic under vsm-discuss, email [hidden email]
To unsubscribe from vsm-discuss, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
Hi Yaguang,

Thanks for the information.  I've gotten a copy of the CRUSH map from one of our POC setups but don't have a copy yet of our 'golden' (production) setup, which I'm told is similar aside from having 2 chassis and a different replication rule being added.  I'm still trying to get some further info but can you tell me for starters if you see anything in here that would prevent our setup from working with VSM?

# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host rack5-storage-1 {
        id -3 # do not change unnecessarily
        # weight 27.920
        alg straw
        hash 0 # rjenkins1
        item osd.1 weight 6.980
        item osd.3 weight 6.980
        item osd.5 weight 6.980
        item osd.6 weight 6.980
}
host rack5-storage-2 {
        id -2 # do not change unnecessarily
        # weight 27.920
        alg straw
        hash 0 # rjenkins1
        item osd.0 weight 6.980
        item osd.2 weight 6.980
        item osd.4 weight 6.980
        item osd.7 weight 6.980
}
chassis IF100-1 {
        id -4 # do not change unnecessarily
        # weight 55.840
        alg straw
        hash 0 # rjenkins1
        item rack5-storage-1 weight 27.920
        item rack5-storage-2 weight 27.920
}
root default {
        id -1 # do not change unnecessarily
        # weight 55.840
        alg straw
        hash 0 # rjenkins1
        item IF100-1 weight 55.840
}

# rules
rule replicated_ruleset {
        ruleset 0
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type host
        step emit
}

# end crush map
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
This is a crush map from our final config:

# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1

# devices
device 0 osd.0
device 1 device1
device 2 device2
device 3 device3
device 4 device4
device 5 device5
device 6 device6
device 7 device7
device 8 device8
device 9 osd.9
device 10 osd.10
device 11 osd.11
device 12 osd.12
device 13 osd.13
device 14 osd.14
device 15 osd.15
device 16 osd.16
device 17 osd.17
device 18 osd.18
device 19 osd.19
device 20 osd.20
device 21 osd.21
device 22 osd.22
device 23 osd.23
device 24 osd.24
device 25 osd.25
device 26 osd.26
device 27 osd.27
device 28 osd.28
device 29 osd.29
device 30 osd.30
device 31 osd.31
device 32 osd.32
device 33 osd.33
device 34 osd.34
device 35 osd.35
device 36 osd.36
device 37 osd.37
device 38 osd.38
device 39 osd.39
device 40 osd.40
device 41 osd.41
device 42 osd.42
device 43 osd.43
device 44 osd.44
device 45 osd.45
device 46 osd.46
device 47 osd.47
device 48 osd.48
device 49 osd.49
device 50 osd.50
device 51 osd.51
device 52 osd.52
device 53 osd.53
device 54 osd.54
device 55 osd.55

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host rack1-storage-1 {
        id -2 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.48 weight 6.840
        item osd.49 weight 6.840
        item osd.50 weight 6.840
        item osd.51 weight 6.840
        item osd.52 weight 6.840
        item osd.53 weight 6.840
        item osd.54 weight 6.840
        item osd.55 weight 6.840
}
host rack1-storage-3 {
        id -3 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.0 weight 6.840
        item osd.9 weight 6.840
        item osd.10 weight 6.840
        item osd.11 weight 6.840
        item osd.12 weight 6.840
        item osd.13 weight 6.840
        item osd.14 weight 6.840
        item osd.15 weight 6.840
}
chassis chassis1 {
        id -8 # do not change unnecessarily
        # weight 109.440
        alg straw
        hash 0 # rjenkins1
        item rack1-storage-1 weight 54.720
        item rack1-storage-3 weight 54.720
}
host rack1-storage-5 {
        id -4 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.16 weight 6.840
        item osd.17 weight 6.840
        item osd.18 weight 6.840
        item osd.19 weight 6.840
        item osd.20 weight 6.840
        item osd.21 weight 6.840
        item osd.22 weight 6.840
        item osd.23 weight 6.840
}
host rack1-storage-6 {
        id -5 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.24 weight 6.840
        item osd.25 weight 6.840
        item osd.26 weight 6.840
        item osd.27 weight 6.840
        item osd.28 weight 6.840
        item osd.29 weight 6.840
        item osd.30 weight 6.840
        item osd.31 weight 6.840
}
chassis chassis2 {
        id -9 # do not change unnecessarily
        # weight 109.440
        alg straw
        hash 0 # rjenkins1
        item rack1-storage-5 weight 54.720
        item rack1-storage-6 weight 54.720
}
host rack1-storage-7 {
        id -6 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.32 weight 6.840
        item osd.33 weight 6.840
        item osd.34 weight 6.840
        item osd.35 weight 6.840
        item osd.36 weight 6.840
        item osd.37 weight 6.840
        item osd.38 weight 6.840
        item osd.39 weight 6.840
}
host rack1-storage-8 {
        id -7 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.40 weight 6.840
        item osd.41 weight 6.840
        item osd.42 weight 6.840
        item osd.43 weight 6.840
        item osd.44 weight 6.840
        item osd.45 weight 6.840
        item osd.46 weight 6.840
        item osd.47 weight 6.840
}
chassis chassis3 {
        id -10 # do not change unnecessarily
        # weight 109.440
        alg straw
        hash 0 # rjenkins1
        item rack1-storage-7 weight 54.720
        item rack1-storage-8 weight 54.720
}
root default {
        id -1 # do not change unnecessarily
        # weight 328.320
        alg straw
        hash 0 # rjenkins1
        item chassis1 weight 109.440
        item chassis2 weight 109.440
        item chassis3 weight 109.440
}

# rules
rule replicated_ruleset {
        ruleset 0
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type host
        step emit
}
rule goldenconfig {
        ruleset 1
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type chassis
        step emit
}

# end crush map
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

ywang19
Administrator

In current VSM, the assumed hierarchy is: root -> storage group -> zone -> host -> osd, and your case is root -> chassis -> host -> osd, and we assume one rule for each storage group, and also there is a corresponding bucket name defined. To support existing cluster, we loose the constraints to
i) the lowest buckets are host->osd,
ii) it’s OK a rule and the corresponding bucket name in “take” has no connection

Generally, your crushmap seems not special, it should be supported in upcoming feature.


BTW, here is one crushmap generated by VSM for your reference:

# begin crush map

tunable choose_local_tries 0

tunable choose_total_tries 50

tunable chooseleaf_descend_once 1

tunable chooseleaf_vary_r 1

 

# devices

device 0 osd.0

device 1 osd.1

device 2 osd.2

device 3 osd.3

device 4 osd.4

device 5 osd.5

device 6 osd.6

device 7 osd.7

device 8 osd.8

 

# types

type 0 osd

type 1 host

type 2 zone

type 3 storage_group

type 4 root

 

# buckets

host vsm-node2_performance_zone_one {

            id -1                  # do not change unnecessarily

            # weight 3.000

            alg straw

            hash 0   # rjenkins1

            item osd.0 weight 1.000

            item osd.1 weight 1.000

            item osd.2 weight 1.000

}

host vsm-node2_high_performance_zone_one {

            id -2                  # do not change unnecessarily

            # weight 0.000

            alg straw

            hash 0   # rjenkins1

}

host vsm-node2_capacity_zone_one {

            id -3                  # do not change unnecessarily

            # weight 0.000

            alg straw

            hash 0   # rjenkins1

}

host vsm-node1_performance_zone_one {

            id -4                  # do not change unnecessarily

            # weight 3.000

            alg straw

            hash 0   # rjenkins1

            item osd.3 weight 1.000

            item osd.4 weight 1.000

            item osd.5 weight 1.000

}

host vsm-node1_high_performance_zone_one {

            id -5                  # do not change unnecessarily

            # weight 0.000

            alg straw

            hash 0   # rjenkins1

}

host vsm-node1_capacity_zone_one {

            id -6                  # do not change unnecessarily

            # weight 0.000

            alg straw

            hash 0   # rjenkins1

}

host vsm-node3_performance_zone_one {

            id -7                  # do not change unnecessarily

            # weight 3.000

            alg straw

            hash 0   # rjenkins1

            item osd.6 weight 1.000

            item osd.7 weight 1.000

            item osd.8 weight 1.000

}

host vsm-node3_high_performance_zone_one {

            id -8                  # do not change unnecessarily

            # weight 0.000

            alg straw

            hash 0   # rjenkins1

}

host vsm-node3_capacity_zone_one {

            id -9                  # do not change unnecessarily

            # weight 0.000

            alg straw

            hash 0   # rjenkins1

}

zone zone_one_performance {

            id -10                # do not change unnecessarily

            # weight 9.000

            alg straw

            hash 0   # rjenkins1

            item vsm-node2_performance_zone_one weight 3.000

            item vsm-node1_performance_zone_one weight 3.000

            item vsm-node3_performance_zone_one weight 3.000

}

zone zone_one_high_performance {

            id -11                # do not change unnecessarily

            # weight 0.300

            alg straw

            hash 0   # rjenkins1

            item vsm-node2_high_performance_zone_one weight 0.100

            item vsm-node1_high_performance_zone_one weight 0.100

            item vsm-node3_high_performance_zone_one weight 0.100

}

zone zone_one_capacity {

            id -12                # do not change unnecessarily

            # weight 0.300

            alg straw

            hash 0   # rjenkins1

            item vsm-node2_capacity_zone_one weight 0.100

            item vsm-node1_capacity_zone_one weight 0.100

            item vsm-node3_capacity_zone_one weight 0.100

}

storage_group performance {

            id -13                # do not change unnecessarily

            # weight 9.000

            alg straw

            hash 0   # rjenkins1

            item zone_one_performance weight 9.000

}

storage_group high_performance {

            id -14                # do not change unnecessarily

            # weight 0.300

            alg straw

            hash 0   # rjenkins1

            item zone_one_high_performance weight 0.300

}

storage_group capacity {

            id -15                # do not change unnecessarily

            # weight 0.300

            alg straw

            hash 0   # rjenkins1

            item zone_one_capacity weight 0.300

}

root vsm {

            id -16                # do not change unnecessarily

            # weight 9.600

            alg straw

            hash 0   # rjenkins1

            item performance weight 9.000

            item high_performance weight 0.300

            item capacity weight 0.300

}

 

# rules

rule performance {

            ruleset 0

            type replicated

            min_size 0

            max_size 10

            step take performance

            step chooseleaf firstn 0 type host

            step emit

}

rule capacity {

            ruleset 1

            type replicated

            min_size 0

            max_size 10

            step take capacity

            step chooseleaf firstn 0 type hostd

            step emit

}

rule high_performance {

            ruleset 2

            type replicated

            min_size 0

            max_size 10

            step take high_performance

            step chooseleaf firstn 0 type host

            step emit

}

 

# end crush map

 


From: rdowell [via vsm-discuss] [ml-node+[hidden email]]
Sent: Tuesday, October 20, 2015 4:36
To: Wang, Yaguang
Subject: RE: Manual deployment without Ceph

This is a crush map from our final config:

# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1

# devices
device 0 osd.0
device 1 device1
device 2 device2
device 3 device3
device 4 device4
device 5 device5
device 6 device6
device 7 device7
device 8 device8
device 9 osd.9
device 10 osd.10
device 11 osd.11
device 12 osd.12
device 13 osd.13
device 14 osd.14
device 15 osd.15
device 16 osd.16
device 17 osd.17
device 18 osd.18
device 19 osd.19
device 20 osd.20
device 21 osd.21
device 22 osd.22
device 23 osd.23
device 24 osd.24
device 25 osd.25
device 26 osd.26
device 27 osd.27
device 28 osd.28
device 29 osd.29
device 30 osd.30
device 31 osd.31
device 32 osd.32
device 33 osd.33
device 34 osd.34
device 35 osd.35
device 36 osd.36
device 37 osd.37
device 38 osd.38
device 39 osd.39
device 40 osd.40
device 41 osd.41
device 42 osd.42
device 43 osd.43
device 44 osd.44
device 45 osd.45
device 46 osd.46
device 47 osd.47
device 48 osd.48
device 49 osd.49
device 50 osd.50
device 51 osd.51
device 52 osd.52
device 53 osd.53
device 54 osd.54
device 55 osd.55

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host rack1-storage-1 {
        id -2 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.48 weight 6.840
        item osd.49 weight 6.840
        item osd.50 weight 6.840
        item osd.51 weight 6.840
        item osd.52 weight 6.840
        item osd.53 weight 6.840
        item osd.54 weight 6.840
        item osd.55 weight 6.840
}
host rack1-storage-3 {
        id -3 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.0 weight 6.840
        item osd.9 weight 6.840
        item osd.10 weight 6.840
        item osd.11 weight 6.840
        item osd.12 weight 6.840
        item osd.13 weight 6.840
        item osd.14 weight 6.840
        item osd.15 weight 6.840
}
chassis chassis1 {
        id -8 # do not change unnecessarily
        # weight 109.440
        alg straw
        hash 0 # rjenkins1
        item rack1-storage-1 weight 54.720
        item rack1-storage-3 weight 54.720
}
host rack1-storage-5 {
        id -4 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.16 weight 6.840
        item osd.17 weight 6.840
        item osd.18 weight 6.840
        item osd.19 weight 6.840
        item osd.20 weight 6.840
        item osd.21 weight 6.840
        item osd.22 weight 6.840
        item osd.23 weight 6.840
}
host rack1-storage-6 {
        id -5 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.24 weight 6.840
        item osd.25 weight 6.840
        item osd.26 weight 6.840
        item osd.27 weight 6.840
        item osd.28 weight 6.840
        item osd.29 weight 6.840
        item osd.30 weight 6.840
        item osd.31 weight 6.840
}
chassis chassis2 {
        id -9 # do not change unnecessarily
        # weight 109.440
        alg straw
        hash 0 # rjenkins1
        item rack1-storage-5 weight 54.720
        item rack1-storage-6 weight 54.720
}
host rack1-storage-7 {
        id -6 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.32 weight 6.840
        item osd.33 weight 6.840
        item osd.34 weight 6.840
        item osd.35 weight 6.840
        item osd.36 weight 6.840
        item osd.37 weight 6.840
        item osd.38 weight 6.840
        item osd.39 weight 6.840
}
host rack1-storage-8 {
        id -7 # do not change unnecessarily
        # weight 54.720
        alg straw
        hash 0 # rjenkins1
        item osd.40 weight 6.840
        item osd.41 weight 6.840
        item osd.42 weight 6.840
        item osd.43 weight 6.840
        item osd.44 weight 6.840
        item osd.45 weight 6.840
        item osd.46 weight 6.840
        item osd.47 weight 6.840
}
chassis chassis3 {
        id -10 # do not change unnecessarily
        # weight 109.440
        alg straw
        hash 0 # rjenkins1
        item rack1-storage-7 weight 54.720
        item rack1-storage-8 weight 54.720
}
root default {
        id -1 # do not change unnecessarily
        # weight 328.320
        alg straw
        hash 0 # rjenkins1
        item chassis1 weight 109.440
        item chassis2 weight 109.440
        item chassis3 weight 109.440
}

# rules
rule replicated_ruleset {
        ruleset 0
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type host
        step emit
}
rule goldenconfig {
        ruleset 1
        type replicated
        min_size 1
        max_size 10
        step take default
        step chooseleaf firstn 0 type chassis
        step emit
}

# end crush map

 


If you reply to this email, your message will be added to the discussion below:

http://vsm-discuss.33411.n7.nabble.com/Manual-deployment-without-Ceph-tp161p205.html

To start a new topic under vsm-discuss, email ml-node+[hidden email]
To unsubscribe from vsm-discuss, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
Thanks for the reply.  If we want to use your earlier suggestion of deploying the current version of VSM on top of our cluster and then enabling the hidden 'Integrate Cluster' option, I have a couple other questions:

1) Should we use 'zone' instead of 'chassis' in our CRUSH map?  The terminology here doesn't really matter for us, it just matched the physical layout of the appliance and we can change it to zone to make it match what VSM uses
2) Do we need to add 'storage group' in our CRUSH map?  The VSM GUI seems to depend on 'storage group' in several places.  I know when creating the manifests I can just set all of our storage into a single group and it should work fine there but I'm wondering if it just needs to be present in the manifest or in the CRUSH map as well in order for the GUI to recognize it as usable storage.
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
Hi Yaguang,

Today I was able to get VSM to recognize our previously-created cluster after finding and fixing a bug in the javascript for the 'integrate cluster' function.  I'd be happy to create and submit a patch for the issue I found but I wanted to check whether you thought it would be worth doing so, since the feature is not technically supported yet and is hidden by default.
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

ywang19
Administrator

Glad to hear you make VSM to recognize previously-created cluster, we really tried the case before. I’m happy you caught out a bug and give a pull request, just make your pull request to master branch is fine. We are also working on another bigger scenario to support clusters with arbitrary crushmap, I hope to discuss this on Nov. VCS meeting.

 

 

From: rdowell [via vsm-discuss] [mailto:ml-node+[hidden email]]
Sent: Thursday, October 29, 2015 6:45 AM
To: Wang, Yaguang
Subject: RE: Manual deployment without Ceph

 

Hi Yaguang,

Today I was able to get VSM to recognize our previously-created cluster after finding and fixing a bug in the javascript for the 'integrate cluster' function.  I'd be happy to create and submit a patch for the issue I found but I wanted to check whether you thought it would be worth doing so, since the feature is not technically supported yet and is hidden by default.


If you reply to this email, your message will be added to the discussion below:

http://vsm-discuss.33411.n7.nabble.com/Manual-deployment-without-Ceph-tp161p227.html

To start a new topic under vsm-discuss, email [hidden email]
To unsubscribe from vsm-discuss, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

ywang19
Administrator

Thanks. I’m waiting for your pull request.

 

From: rdowell [via vsm-discuss] [mailto:ml-node+[hidden email]]
Sent: Friday, October 30, 2015 6:46 AM
To: Wang, Yaguang
Subject: RE: Manual deployment without Ceph

 

Created a new issue https://01.org/jira/browse/VSM-375


If you reply to this email, your message will be added to the discussion below:

http://vsm-discuss.33411.n7.nabble.com/Manual-deployment-without-Ceph-tp161p230.html

To start a new topic under vsm-discuss, email [hidden email]
To unsubscribe from vsm-discuss, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
In reply to this post by ywang19
Yaguang,

We're seeing a strange result when attempting to import a cluster into VSM using 'Integrate Cluster'.  The operation seems to work fine as there are no errors reported in the GUI or the logs and the majority of the ceph cluster info correctly appears in the VSM GUI.  The weird behavior we're seeing is specifically on the 'Manage Devices' tab in VSM.

I created a simple cluster with 3 nodes and 1 OSD per node using a testing setup that is simpler than our intended production setup which I gave you the crush map for previously.  One OSD imports correctly, but the other 2 seem to only partially succeed - they show up in 'Manage Devices' but show 'Internal Server Error' when I try to look at the detail view for either the data or journal device.  In addition, a duplicate OSD also shows up in 'Manage Devices', always named 'osd.x' and while this device shows as 'Uninitialized', it DOES show the correct device info in the detail view.

I've attached several screenshots below, one just after deploying our cluster and then installing VSM on top of it, but before running 'Integrate Cluster', and the rest after invoking the 'Integrate Cluster' operation.

Before Integrating - VSM is showing all cluster nodes and 3 uninitialized OSD's.  At this point, I assume VSM is only displaying the info that is present in the installation manifest files, since if this were a regular VSM installation, this would also be prior to invoking 'Create Cluster' so the Ceph cluster would not actually be up and running and none of the OSD's would be running yet.
Before Integrate cluster

After Integrating - the 3 initialized OSD's are present, but 2 of the uninitialized ones remain, and the corresponding real OSD's are showing 'Internal Server Error' in the device details:
After Integrating

Data device status - osd.2:
Data device status - osd.2

Data device status - osd.x:
Data device status - osd.x

The CRUSH map seems normal:
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable straw_calc_version 1

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host rdowell-vsm-node3 {
	id -2		# do not change unnecessarily
	# weight 0.010
	alg straw
	hash 0	# rjenkins1
	item osd.0 weight 0.010
}
host rdowell-vsm-node1 {
	id -3		# do not change unnecessarily
	# weight 0.010
	alg straw
	hash 0	# rjenkins1
	item osd.1 weight 0.010
}
host rdowell-vsm-node2 {
	id -4		# do not change unnecessarily
	# weight 0.010
	alg straw
	hash 0	# rjenkins1
	item osd.2 weight 0.010
}
root default {
	id -1		# do not change unnecessarily
	# weight 0.030
	alg straw
	hash 0	# rjenkins1
	item rdowell-vsm-node3 weight 0.010
	item rdowell-vsm-node1 weight 0.010
	item rdowell-vsm-node2 weight 0.010
}

# rules
rule replicated_ruleset {
	ruleset 0
	type replicated
	min_size 1
	max_size 10
	step take default
	step chooseleaf firstn 0 type host
	step emit
}

# end crush map
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
One thing I noted - I dug pretty deep into VSM to try to figure out exactly where the work of the 'integrate cluster' function is executed.  It appears as though it relies on the ceph-mds service.  Our normal cluster install doesn't start this service so it isn't running when the 'integrate cluster' function is invoked.  Would starting ceph-mds on the cluster nodes allow integrate to work properly?
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

rdowell
Yaguang,

I saw your response to John on the other thread he created, so we'll start looking at the WIP-53 branch instead.  Thanks for the info, I had essentially reached the same conclusion that the feature on the master branch was incomplete, knowing that there is an active development branch is very helpful.
Reply | Threaded
Open this post in threaded view
|

RE: Manual deployment without Ceph

ywang19
Administrator

We have some initial tests with WIP-53, can import some existing cluster and create pools from VSM. the wip-53 is a temporary, at some point when the feature is roughly stable, I will merge it into master to continue 2.1 development.

 

 

From: rdowell [via vsm-discuss] [mailto:ml-node+[hidden email]]
Sent: Thursday, November 19, 2015 5:41 AM
To: Wang, Yaguang
Subject: RE: Manual deployment without Ceph

 

Yaguang,

I saw your response to John on the other thread he created, so we'll start looking at the WIP-53 branch instead.  Thanks for the info, I had essentially reached the same conclusion that the feature on the master branch was incomplete, knowing that there is an active development branch is very helpful.


If you reply to this email, your message will be added to the discussion below:

http://vsm-discuss.33411.n7.nabble.com/Manual-deployment-without-Ceph-tp161p247.html

To start a new topic under vsm-discuss, email [hidden email]
To unsubscribe from vsm-discuss, click here.
NAML