add/remove osd/mon out of band

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

add/remove osd/mon out of band

jcalcote
Hi Yaguang,

How does VSM react when a user deploys VSM (and VSM deploys Ceph), and the user then uses a ceph command-line tool to remove or add an OSD or Monitor?

I ask because people who import an existing cluster into VSM (like we do) will expect to be able to continue to use their usual command-line tools to manage their Ceph cluster if they want to. I suspect there is going to be a desire to have VSM be a bit more flexible with out-of-band cluster changes.

I suggest it might be a good plan to begin working on a background Ceph query that VSM can do once every 30-60 seconds to check the cluster topology in order to sync out-of-band changes in OSDs and Monitors - and perhaps even in cluster nodes.

Incidentally, for cluster changes (add/remove server nodes), we currently literally revert the mysql database to a clean state and reimport the cluster after an out-of-band change.

Thanks,
John
Reply | Threaded
Open this post in threaded view
|

RE: add/remove osd/mon out of band

ywang19
Administrator

Hi John,

 

As you expected,  out-of-band cluster change will confuse VSM. but treating the feature to import existing cluster as the starting point, it would be a good plan to re-sync cluster. Clean up database then re-import is an approach, we are also evaluating a feature to un-deploy cluster and re-import, which should hit the same goal.

 

 

-yaguang

 

 

From: jcalcote [via vsm-discuss] [mailto:ml-node+[hidden email]]
Sent: Thursday, January 28, 2016 7:13 AM
To: Wang, Yaguang
Subject: add/remove osd/mon out of band

 

Hi Yaguang,

How does VSM react when a user deploys VSM (and VSM deploys Ceph), and the user then uses a ceph command-line tool to remove or add an OSD or Monitor?

I ask because people who import an existing cluster into VSM (like we do) will expect to be able to continue to use their usual command-line tools to manage their Ceph cluster if they want to. I suspect there is going to be a desire to have VSM be a bit more flexible with out-of-band cluster changes.

I suggest it might be a good plan to begin working on a background Ceph query that VSM can do once every 30-60 seconds to check the cluster topology in order to sync out-of-band changes in OSDs and Monitors - and perhaps even in cluster nodes.

Incidentally, for cluster changes (add/remove server nodes), we currently literally revert the mysql database to a clean state and reimport the cluster after an out-of-band change.

Thanks,
John


If you reply to this email, your message will be added to the discussion below:

http://vsm-discuss.33411.n7.nabble.com/add-remove-osd-mon-out-of-band-tp423.html

To start a new topic under vsm-discuss, email [hidden email]
To unsubscribe from vsm-discuss, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: add/remove osd/mon out of band

jcalcote
Yaguang,

My team has decided we'd like to add code to VSM that periodically scans the Ceph cluster topology and updates the database accordingly in order to allow out-of-band changes to be handled in a reasonable way.

I believe this would be a good feature for VSM for the following reasons:

1. Obviously, out-of-band tools just work - users make command-line changes to the ceph topology and VSM adjusts its view of that topology accordingly. The most significant such out-of-band tools would be ceph utilities themselves. As it stands, VSM is a less-than-ideal management solution because once deployed - especially against an existing imported cluster - users can no longer use the ceph tools they're used to using to make changes to their cluster.

2. VSM management code can be simplified - rather than have each operation (add/remove osd/monitor/server, etc) update the database in a manner that's unique to its purpose, it could just make the changes with ceph utility calls, and then allow VSM to update its own state based on the periodic scan. VSM could also provide a "refresh now" method that allows operations to trigger the update immediately after a known state change. One such call at the end of any call out to ceph to make changes would be sufficient to cause VSM to update itself as per the changes just made.

Questions:

Are you interested in accepting such a change from us?

If so, do you want us to use a "wip" branch to make these changes and just merge in from (or to) master periodically?

Do you have any suggestions about where such a periodic refresh should best exist? I'm really interested in doing this feature correctly so your feedback on this question is critical. While I have a pretty good understanding of VSM architecture at this point, I'm still a bit fuzzy on the exact roles of conductor and scheduler (for instance). Where is the best location to put a periodic scan routine that updates the VSM database on cluster topology?

Thanks,
John
Reply | Threaded
Open this post in threaded view
|

Re: add/remove osd/mon out of band

ywang19
Administrator
Hi John,

I suggest to put them onto a wip branch, for how to implement it, the task could firstly undeploy cluster, then re-import it, which should well utilize current code logic.
Generally, it is a good feature to add, if no relevant broken, I will accept it.
One open is how can we make sure the database is updated, if our operations on VSM use old topo,   There might be some messy issues.

I am on vocation till February 13. the response expects to be slow. Sorry for the inconvenience.

-yaguang

在 2016年2月6日,上午2:26,jcalcote [via vsm-discuss] <[hidden email]<mailto:[hidden email]>> 写道:

Yaguang,

My team has decided we'd like to add code to VSM that periodically scans the Ceph cluster topology and updates the database accordingly in order to allow out-of-band changes to be handled in a reasonable way.

I believe this would be a good feature for VSM for the following reasons:

1. Obviously, out-of-band tools just work - users make command-line changes to the ceph topology and VSM adjusts its view of that topology accordingly. The most significant such out-of-band tools would be ceph utilities themselves. As it stands, VSM is a less-than-ideal management solution because once deployed - especially against an existing imported cluster - users can no longer use the ceph tools they're used to using to make changes to their cluster.

2. VSM management code can be simplified - rather than have each operation (add/remove osd/monitor/server, etc) update the database in a manner that's unique to its purpose, it could just make the changes with ceph utility calls, and then allow VSM to update its own state based on the periodic scan. VSM could also provide a "refresh now" method that allows operations to trigger the update immediately after a known state change. One such call at the end of any call out to ceph to make changes would be sufficient to cause VSM to update itself as per the changes just made.

Questions:

Are you interested in accepting such a change from us?

If so, do you want us to use a "wip" branch to make these changes and just merge in from (or to) master periodically?

Do you have any suggestions about where such a periodic refresh should best exist? I'm really interested in doing this feature correctly so your feedback on this question is critical. While I have a pretty good understanding of VSM architecture at this point, I'm still a bit fuzzy on the exact roles of conductor and scheduler (for instance). Where is the best location to put a periodic scan routine that updates the VSM database on cluster topology?

Thanks,
John

________________________________
If you reply to this email, your message will be added to the discussion below:
http://vsm-discuss.33411.n7.nabble.com/add-remove-osd-mon-out-of-band-tp423p436.html
To start a new topic under vsm-discuss, email [hidden email]<mailto:[hidden email]>
To unsubscribe from vsm-discuss, click here<
NAML<
http://vsm-discuss.33411.n7.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
Reply | Threaded
Open this post in threaded view
|

Re: add/remove osd/mon out of band

jcalcote
VSM-431 created and assigned to myself. Please update as necessary.